diff --git a/CHANGELOG.md b/CHANGELOG.md index e2873689..bad62a54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **129×129 lid-driven cavity validation recorded** — the AVX2, OpenMP and CUDA projection + backends match Ghia et al. (1982) at 129×129 with RMS_u / RMS_v of 0.0017 / 0.0024 + (Re=100), 0.0096 / 0.0328 (Re=400) and 0.0299 / 0.0300 (Re=1000), identical across + backends to four decimals and across three EC2 workflow runs. Each backend line of + `test_cavity_backends` now also prints the steps actually run, whether the kinetic-energy + residual stopped the run early, and that residual + (`docs/validation/cavity-backends-validation.md`, `tests/validation/test_cavity_backends.c`). - **Multigrid pressure solve on the OpenMP projection solver** — `projection_omp` now accepts `NS_PRESSURE_SOLVER_MULTIGRID` (OpenMP multigrid V-cycles, RHS interior mean subtracted for Neumann compatibility) and @@ -108,6 +115,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- `scripts/ec2-validate.sh` runs every `CavityBackend_*` ctest entry. It ran the test binary + without a filter, which skips the Re=400 and Re=1000 cases, and under `set -e` a failing run + ended the script before its FAILED summary. - `poisson_solve_3d()` now checks `poisson_solver_init()`'s return status: a failed init (e.g. multigrid on non-2^k+1 dims) no longer leaves a broken solver in the convenience cache; the call returns -1 and later valid calls re-create the solver. diff --git a/ROADMAP.md b/ROADMAP.md index ff5d8eeb..455d7641 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -455,10 +455,15 @@ headers. See CHANGELOG. Lid-driven cavity (33×33, Re=100, all backends), Taylor-Green vortex, and Poiseuille flow are all validated. See CHANGELOG and `docs/validation/`. -**Still needed — 129×129 release validation** (too slow for CI; uses `CAVITY_FULL_VALIDATION=1`): +**129×129 release validation** (`-DCAVITY_FULL_VALIDATION=ON`; runs in the EC2 GPU workflow on +every push to master, about 50 minutes): -- [ ] Full Ghia validation at 129×129 for Re=100, 400, 1000 (record RMS on EC2) +- [x] Full Ghia validation at 129×129 for Re=100, 400, 1000 — AVX2/OMP/GPU projection RMS + ≤ 0.033, recorded in `docs/validation/cavity-backends-validation.md` - [ ] Extended cavity convergence to true steady-state (residual < 1e-8) +- [ ] Explicit Euler cavity cases stop at 11,300–11,800 steps (t ≈ 1.1–1.2) through the harness's + kinetic-energy exit, on the 33×33 CI grid (of 25,000) and at 129×129 (of 250,000) alike; make + them run to a developed flow or drop them from Ghia validation - [ ] Multi-Reynolds grid-convergence study (Richardson extrapolation) - [ ] Extended-time Taylor-Green decay-rate verification - [ ] Cross-architecture consistency (all backends identical within 0.1%) @@ -473,7 +478,7 @@ all validated. See CHANGELOG and `docs/validation/`. | Test | CI Mode | Release Mode | |------|---------|--------------| -| Cavity Ghia Validation | 33×33, 5000 steps | 129×129, 50000 steps | +| Cavity Ghia Validation | 33×33, 5000 steps | 129×129; 50000 steps (Re=100), 60000 (Re=400), 100000 (Re=1000) | | Cavity Re=400 Stability | 25×25, 500 steps | 65×65, 20000 steps | | Grid Convergence | 17→25→33 | 33→65→129 | | Taylor-Green Vortex | 32×32, 200 steps | 128×128, 10000 steps | diff --git a/docs/reference/solvers.md b/docs/reference/solvers.md index e44ebd77..c91633a7 100644 --- a/docs/reference/solvers.md +++ b/docs/reference/solvers.md @@ -641,11 +641,13 @@ Classic benchmark problem validated against Ghia et al. (1982): - All other walls no-slip - Reynolds number Re = ρUL/μ -**Results (Re=100, 129×129 grid):** -- Centerline velocity profiles match published data -- RMS error < 0.01 +**Results (129×129 grid, AVX2/OpenMP/CUDA projection):** +- Centerline velocity profiles match published data at Re = 100, 400 and 1000 +- RMS error vs Ghia (u / v): 0.0017 / 0.0024 at Re=100, 0.0096 / 0.0328 at Re=400, + 0.0299 / 0.0300 at Re=1000 -See [validation/lid-driven-cavity.md](validation/lid-driven-cavity.md) for details. +See [lid-driven-cavity.md](../validation/lid-driven-cavity.md) and +[cavity-backends-validation.md](../validation/cavity-backends-validation.md) for details. ### Taylor-Green Vortex diff --git a/docs/validation/cavity-backends-validation.md b/docs/validation/cavity-backends-validation.md index 76c2112d..d6939c60 100644 --- a/docs/validation/cavity-backends-validation.md +++ b/docs/validation/cavity-backends-validation.md @@ -16,7 +16,7 @@ This document describes the comprehensive backend validation system for the lid- - **CPU Scalar** (`projection`) - **AVX2/SIMD** (`projection_optimized`) - **OpenMP** (`projection_omp`) -- **CUDA GPU** (`projection_jacobi_gpu`) +- **CUDA GPU** (`projection_gpu`) ### Explicit Euler Backends - **CPU** (`explicit_euler`) @@ -82,18 +82,23 @@ dt: 0.0005 ``` ### Full Validation Mode -```c -#define CAVITY_FULL_VALIDATION 1 +Enabled with the CMake option `-DCAVITY_FULL_VALIDATION=ON`: + +```text Re=100: Grid 129×129, 50000 steps (250000 for Explicit Euler), dt 0.0002 Re=400: Grid 129×129, 60000 steps (projection only), dt 0.0005 Re=1000: Grid 129×129, 100000 steps (projection only), dt 0.0005 ``` +Step counts are budgets: the harness stops a run early once the relative change in kinetic +energy per step drops below 1e-8 (after step 100), and each backend prints a `Steps run:` line +with the steps it actually ran. + Higher-Reynolds cases run the **projection backends only** (the production solver) and require the finer 129×129 grid to resolve the stronger primary vortex and the -secondary corner vortices. The Re=400/1000 `dt`/step counts are starting points — -refine them after recording the first full-validation RMS values (see below). +secondary corner vortices. Every projection case passes with at least a 3× margin on the +RMS target (see Full Validation Results below), so the step budgets are unchanged. To run full validation: ```bash @@ -136,8 +141,8 @@ All available backends produce consistent results (within 0.1%) ## Key Findings -1. **Projection method meets scientific target** (RMS < 0.10) ✅ -2. **Explicit Euler meets relaxed target** (RMS < 0.15) ✅ +1. **Projection method meets scientific target** (RMS < 0.10) ✅ — at 33×33, and at 129×129 for Re=100, 400 and 1000 +2. **Explicit Euler meets relaxed target** (RMS < 0.15) ✅ — but its 129×129 runs stop early (see Known Issue below) 3. **CPU and OpenMP backends are consistent** ✅ 4. **Test correctly fails when RMS >= target** ✅ @@ -176,33 +181,68 @@ To exclude validation tests (e.g., sanitizer builds): ctest -LE validation ``` +Full-validation builds (`-DCAVITY_FULL_VALIDATION=ON`) replace `CavityBackendsTest` with one +ctest entry per backend and Reynolds number (`CavityBackend_*`, registered in `CMakeLists.txt`). +The Cross-Architecture Validation (EC2) workflow, `.github/workflows/gpu-validation.yml`, builds +that configuration on every push to master and on pull requests labelled `gpu-test`. It runs the +`validation` label on a g4dn.4xlarge with an NVIDIA T4 and writes a per-(Re, backend) RMS table +to the job summary. `scripts/ec2-validate.sh` runs the same `CavityBackend_*` entries on a +manually started instance. + ## Full Validation Results (129×129) -Record the RMS values from the `-DCAVITY_FULL_VALIDATION=ON` run here. All entries -must satisfy RMS < 0.10 (projection). Fill in once the EC2/GPU run completes. - -| Re | Backend | RMS_u | RMS_v | Pass? | -|------|---------------|--------|--------|-------| -| 100 | AVX2/SIMD | _TBD_ | _TBD_ | _TBD_ | -| 100 | OpenMP | _TBD_ | _TBD_ | _TBD_ | -| 100 | CUDA GPU | _TBD_ | _TBD_ | _TBD_ | -| 400 | AVX2/SIMD | _TBD_ | _TBD_ | _TBD_ | -| 400 | OpenMP | _TBD_ | _TBD_ | _TBD_ | -| 400 | CUDA GPU | _TBD_ | _TBD_ | _TBD_ | -| 1000 | AVX2/SIMD | _TBD_ | _TBD_ | _TBD_ | -| 1000 | OpenMP | _TBD_ | _TBD_ | _TBD_ | -| 1000 | CUDA GPU | _TBD_ | _TBD_ | _TBD_ | - -> If Re=1000 does not converge to RMS < 0.10 within the step budget, increase -> `VALIDATION_STEPS_RE1000` (more physical time) before relaxing the tolerance — -> the target is a scientific standard, not a tunable. +All nine projection cases pass the RMS < 0.10 target with at least a 3× margin, and the AVX2, +OpenMP and CUDA backends give the same RMS to four decimals at every Reynolds number. + +| Re | Backend | RMS_u | RMS_v | Pass? | Step budget × dt (t) | Wall time | +|------|-----------|--------|--------|-------|----------------------|-----------| +| 100 | AVX2/SIMD | 0.0017 | 0.0024 | Yes | 50000 × 0.0002 (10) | 293 s | +| 100 | OpenMP | 0.0017 | 0.0024 | Yes | 50000 × 0.0002 (10) | 428 s | +| 100 | CUDA GPU | 0.0017 | 0.0024 | Yes | 50000 × 0.0002 (10) | 516 s | +| 400 | AVX2/SIMD | 0.0096 | 0.0328 | Yes | 60000 × 0.0005 (30) | 336 s | +| 400 | OpenMP | 0.0096 | 0.0328 | Yes | 60000 × 0.0005 (30) | 487 s | +| 400 | CUDA GPU | 0.0096 | 0.0328 | Yes | 60000 × 0.0005 (30) | 481 s | +| 1000 | AVX2/SIMD | 0.0299 | 0.0300 | Yes | 100000 × 0.0005 (50) | 516 s | +| 1000 | OpenMP | 0.0299 | 0.0300 | Yes | 100000 × 0.0005 (50) | 801 s | +| 1000 | CUDA GPU | 0.0299 | 0.0300 | Yes | 100000 × 0.0005 (50) | 805 s | + +**Source:** GitHub Actions run 34853615452 of the Cross-Architecture Validation (EC2) workflow, +commit `6e1b104` (2026-09-14). Runs 34815324268 (`23b196a`) and 30116961115 (2026-07-24, #201) +produced the same RMS values. Configuration: g4dn.4xlarge (16 vCPUs, NVIDIA T4), gcc Release +with AVX2 and CUDA enabled, `ctest -L validation -j 4` with `OMP_NUM_THREADS=4`. Wall times come +from that shared run, with several tests running at once, so they are not performance +benchmarks. These runs predate the `Steps run:` line, so the table lists step budgets; the next +workflow run reports the steps each case ran. + +> If a later change pushes a case above RMS 0.10, increase its step budget (more physical +> time) before relaxing the tolerance — the target is a scientific standard, not a tunable. + +### Known Issue: Explicit Euler at 129×129 + +The Explicit Euler cases are not evidence of 129×129 accuracy. Both solvers cap dt at 1e-4, and +the harness's kinetic-energy exit (relative change per step below 1e-8) ends the runs long before +their 250,000-step budget: + +| Backend | Steps run | Simulated time | KE residual | RMS_u | RMS_v | +|-----------|-----------|----------------|-------------|--------|--------| +| AVX2/SIMD | 11,323 | ≈ 1.1 | 8.3e-9 | 0.0957 | 0.1293 | +| OpenMP | 11,775 | ≈ 1.2 | 9.8e-9 | 0.0957 | 0.1277 | + +Measured on a local Windows (MSVC Release) build with 4 OpenMP threads; the RMS values match the +EC2 runs. The 33×33 CI case stops the same way: the scalar Euler solver ends at 11,300 of 25,000 +steps with RMS_u 0.0957 and RMS_v 0.1284. The flow needs about 10–20 time units to reach steady +state at Re=100 (see [lid-driven-cavity.md](lid-driven-cavity.md)), so these runs pass the 0.15 +target without a developed solution on either grid. The test logic is unchanged; the follow-up is +tracked in ROADMAP §6.1. ## Next Steps ### Immediate (for v1.0) -1. Run full 129×129 validation (release mode) and fill in the results table above -2. Tune Re=400/1000 `dt`/step counts if convergence is incomplete -3. Tick the ROADMAP 6.1.1 / 6.1.5 boxes once recorded +1. ROADMAP §6.1 cross-architecture consistency stays open: `CavityBackend_Consistency` compares + CPU, AVX2 and OpenMP center values at 33×33 without the GPU, and matching RMS values at + 129×129 are not a field-level 0.1% comparison. +2. Make the Explicit Euler 129×129 cases run to a developed flow, or drop them from full + validation (see Known Issue above). ### Future Enhancements 1. Add grid convergence study (33→65→129→257) diff --git a/docs/validation/lid-driven-cavity.md b/docs/validation/lid-driven-cavity.md index 8f732598..debe1b05 100644 --- a/docs/validation/lid-driven-cavity.md +++ b/docs/validation/lid-driven-cavity.md @@ -132,35 +132,45 @@ Computed by: Two modes are available: **Fast Mode (CI):** -- Iterations: 1500-4000 steps +- Steps: 5000 for the 33×33 Ghia comparisons (25000 for the Explicit Euler backend cases) - Time step: 0.0005 - Purpose: Quick regression testing **Full Validation Mode:** -- Iterations: 3000-10000 steps -- Time step: 0.0005 -- Purpose: Comprehensive validation +- Grid: 129×129 at Re = 100, 400 and 1000 +- Steps: 50000 at Re=100 (dt 0.0002), 60000 at Re=400 and 100000 at Re=1000 (dt 0.0005) +- Purpose: Release validation; runs in the EC2 GPU workflow on every push to master -Enable full mode with: -```c -#define CAVITY_FULL_VALIDATION 1 +Enable full mode with the CMake option: +```bash +cmake -B build -DCAVITY_FULL_VALIDATION=ON ``` +See [cavity-backends-validation.md](cavity-backends-validation.md) for the per-backend +parameters and ctest entries. + ## Current Solver Performance -### Status: ACCEPTABLE (Engineering Quality) +### 129×129 Release Validation + +The AVX2, OpenMP and CUDA projection backends match Ghia et al. at 129×129, all under the +0.10 target (RMS_u / RMS_v): Re=100 0.0017 / 0.0024, Re=400 0.0096 / 0.0328, Re=1000 +0.0299 / 0.0300. The full table and run details are in +[cavity-backends-validation.md](cavity-backends-validation.md). + +### 33×33 CI Validation -The current projection solver with CG Poisson achieves: -- **u-centerline RMS: ~0.10** (target: < 0.10 ✅) -- **v-centerline RMS: ~0.08** (target: < 0.10 ✅) -- **Convergence:** Reaches steady state within tolerance -- **Poisson Solver:** Conjugate Gradient (CG) with tolerance 1e-6 +`CavityBackendsTest` runs each projection backend at 33×33 for 5000 steps with dt = 0.0005 +(t = 2.5), each with a CG pressure solve on its own backend. The CPU scalar backend gives: +- **u-centerline RMS: 0.0382** (target: < 0.10 ✅) +- **v-centerline RMS: 0.0440** (target: < 0.10 ✅) +- **Not a steady state:** the run uses its whole 5000-step budget and ends with a kinetic-energy + change of about 2.7e-5 per step, far above the 1e-8 stop threshold. t = 2.5 is also well short + of the 10–20 time units the flow needs (see Insufficient Time Stepping below), so this is a + quick regression check, not a converged comparison. -**Configuration:** -- Grid: 33×33 (current CI tests) -- Time steps: ~3000-4000 iterations -- dt = 0.0005 -- Poisson: CG method (`POISSON_METHOD_CG`) with scalar or SIMD backend (`POISSON_BACKEND_SCALAR` / `POISSON_BACKEND_SIMD`) +See [cavity-backends-validation.md](cavity-backends-validation.md) for the other backends' CI +results. ### Path to Excellence (RMS < 0.05) @@ -197,10 +207,13 @@ The lid-driven cavity flow at Re = 100 requires ~10-20 time units to reach stead - Even 1% pressure error → 5-10% velocity error after many time steps - Insufficient Poisson convergence → spurious divergence → incorrect vortex structure -**Current solution:** All backends now use robust solvers: -- CPU: Conjugate Gradient (CG) - reliable convergence in ~150 iterations -- SIMD/OMP: Red-Black SOR or CG-SIMD -- GPU: Jacobi with sufficient iterations for parallelism +**Current solution:** Every projection backend solves the pressure equation with Conjugate +Gradient (CG) on its own backend (the scalar and OpenMP solvers can switch to multigrid through +`ns_solver_params_t.pressure_solver`): +- CPU: scalar CG - reliable convergence in ~150 iterations +- SIMD: CG with AVX2/NEON kernels +- OMP: OpenMP CG +- GPU: device-resident CUDA CG #### 3. Time Step Selection diff --git a/scripts/ec2-validate.sh b/scripts/ec2-validate.sh index 470b9cc4..bd366a80 100644 --- a/scripts/ec2-validate.sh +++ b/scripts/ec2-validate.sh @@ -1,20 +1,23 @@ #!/usr/bin/env bash -# ec2-validate.sh — Build and run 129x129 cavity validation on EC2 +# ec2-validate.sh — Build and run the 129x129 cavity validation on EC2 +# +# Runs every CavityBackend_* ctest entry (Re=100, 400 and 1000), the same cases the +# Cross-Architecture Validation (EC2) workflow runs on each push to master. # # Quick-start (on a fresh g4dn.4xlarge with NVIDIA Deep Learning AMI): # # # First time: # git clone cfd && cd cfd # ./scripts/ec2-validate.sh --setup -# ./scripts/ec2-validate.sh --build --run +# ./scripts/ec2-validate.sh --build --release --run # # # Subsequent runs (after code changes): # git pull # or: rsync from local machine -# ./scripts/ec2-validate.sh --build --run -# -# # Release mode for benchmarking: # ./scripts/ec2-validate.sh --build --release --run # +# # Debug build (much slower; for stepping through with gdb): +# ./scripts/ec2-validate.sh --build --run +# # VSCode Remote-SSH debugging: # 1. Install "Remote - SSH" extension in VSCode # 2. Ctrl+Shift+P > "Remote-SSH: Connect to Host..." > ubuntu@ @@ -56,10 +59,10 @@ while [[ $# -gt 0 ]]; do echo "" echo " --setup Install build dependencies (first time only)" echo " --build Build the project (Debug mode by default)" - echo " --release Use Release mode instead of Debug" - echo " --run Run the 129x129 cavity validation test" + echo " --release Use Release mode instead of Debug (matches CI)" + echo " --run Run all 129x129 cavity validation cases (Re=100/400/1000)" echo " --all Do setup + build + run (default if no flags given)" - echo " --threads Set OMP_NUM_THREADS (default: auto-detect)" + echo " --threads Set OMP_NUM_THREADS per test (default: nproc / 4)" exit 0 ;; *) echo "Unknown option: $1"; exit 1 ;; @@ -158,14 +161,17 @@ fi # ---------- Run ---------- if [[ "${DO_RUN}" == "true" ]]; then echo "" - echo "=== Running 129x129 Cavity Validation ===" - - if [[ -n "${OMP_THREADS}" ]]; then - export OMP_NUM_THREADS="${OMP_THREADS}" - echo "OMP_NUM_THREADS=${OMP_NUM_THREADS}" - else - echo "OMP_NUM_THREADS=auto ($(nproc) cores available)" + echo "=== Running 129x129 Cavity Validation (Re=100/400/1000) ===" + + # Same scheduling as CI: 4 ctest slots (each CavityBackend_ entry reserves 2) + # and nproc/4 OpenMP threads per test + CTEST_JOBS=4 + if [[ -z "${OMP_THREADS}" ]]; then + OMP_THREADS=$(( $(nproc) / CTEST_JOBS )) + OMP_THREADS=$(( OMP_THREADS > 0 ? OMP_THREADS : 1 )) fi + export OMP_NUM_THREADS="${OMP_THREADS}" + echo "ctest -j ${CTEST_JOBS}, OMP_NUM_THREADS=${OMP_NUM_THREADS} ($(nproc) cores available)" BINARY="${BUILD_DIR}/test_cavity_backends" if [[ ! -f "${BINARY}" ]]; then @@ -173,15 +179,22 @@ if [[ "${DO_RUN}" == "true" ]]; then exit 1 fi + LOG="${BUILD_DIR}/validation_output.txt" echo "" START_TIME=$(date +%s) - "${BINARY}" - EXIT_CODE=$? + # Capture the status instead of letting set -e end the script before the summary + EXIT_CODE=0 + ctest --test-dir "${BUILD_DIR}" -R '^CavityBackend_' --no-tests=error \ + --output-on-failure -V -j "${CTEST_JOBS}" 2>&1 | tee "${LOG}" || EXIT_CODE=$? END_TIME=$(date +%s) ELAPSED=$((END_TIME - START_TIME)) MINUTES=$((ELAPSED / 60)) SECONDS=$((ELAPSED % 60)) + echo "" + echo "=== Results (full log: ${LOG}) ===" + grep -E 'Testing backend|Steps run:|RMS_u:|SKIPPED|\[FAILED\]' "${LOG}" || true + echo "" echo "============================================" if [[ ${EXIT_CODE} -eq 0 ]]; then diff --git a/tests/validation/test_cavity_backends.c b/tests/validation/test_cavity_backends.c index fbb2e397..87f10c72 100644 --- a/tests/validation/test_cavity_backends.c +++ b/tests/validation/test_cavity_backends.c @@ -229,6 +229,11 @@ static void test_backend_validation(const char* solver_type, TEST_ASSERT_NOT_NULL(ctx); + /* Steps actually run: the harness stops early once the KE residual drops below 1e-8 */ + printf(" Steps run: %d/%d stopped by KE threshold: %s KE residual: %.2e\n", + result.steps_completed, max_steps, result.converged ? "yes" : "no", + result.final_residual); + /* Extract centerline profiles */ profile_data_t profiles = extract_profiles_from_ctx(ctx); TEST_ASSERT_NOT_NULL_MESSAGE(profiles.y_coords, "Failed to extract profiles");