Upgrade to C++20, update lc0 integration, and add one-game-per-file output

[ContradNamiseb]

This pull request modernizes the build system and codebase, updates the lc0 integration, and introduces several improvements to training data handling and developer tooling. The main changes include upgrading to C++20, updating the lc0 submodule and source file paths, switching to the V6 training data format, and enhancing the output format to write one game per chunk file. Additionally, new calibration scripts and an absl compatibility header were added.

Build System and Language Modernization

• Upgraded the project to use C++20 (from C++17), set explicit Release build type, simplified include directories, and added lc0/src/utils/string.cc to resolve linker errors. The build now prefers the system zlib on Unix and includes platform-specific compiler flags for better compatibility and legacy code support. (CMakeLists.txt, .github/workflows/cmake-multi-platform.yml) [1] [2] [3]

lc0 Integration Updates

• Updated the lc0 submodule to track the official master branch and adjusted source file paths to match the latest lc0 structure (e.g., removed bitboard.cc, replaced neural/writer.cc with trainingdata/writer.cc). (.gitmodules, lc0) [1] [2]

Training Data Format and Output Improvements

• Upgraded from V4 to V6 training data format, updated related source files and hash utilities, and modified the output logic so each game is written to its own chunk file instead of batching multiple games together. (PULL_REQUEST.md)

Developer Tooling and Scripts

• Added three new Python scripts (calibrate_s.py, calibrate_s2.py, calibrate_s3.py) for analyzing and calibrating the relationship between Q and D values in training data, supporting improved model calibration and validation. (scripts/calibrate_s.py, scripts/calibrate_s2.py, scripts/calibrate_s3.py) [1] [2] [3]

Dependency Compatibility

• Introduced a minimal absl/cleanup/cleanup.h header to satisfy new lc0 dependencies. (absl/cleanup/cleanup.h)## Summary
  This PR modernizes the trainingdata-tool by upgrading to C++20, updating the lc0 integration to work with the latest lc0 codebase, and improving the output format to write one game per chunk file.

Changes

Build System Updates

• Upgraded C++ standard from C++17 to C++20
• Set explicit Release build type
• Simplified include directories
• Added lc0/src/utils/string.cc to fix linker error for StrSplit function

lc0 Integration Updates

• Updated lc0 source file paths to match latest lc0 structure:
  • Removed lc0/src/chess/bitboard.cc (no longer needed)
  • Changed lc0/src/neural/writer.cc to lc0/src/trainingdata/writer.cc
• Updated .gitmodules for lc0 submodule
• Added absl/ library dependency

Training Data Format Upgrade

• Upgraded from V4 to V6 training data format
• Replaced V4TrainingDataHashUtil.h with V6TrainingDataHashUtil.h
• Updated related source files for V6 compatibility

Output Format Improvement

• Modified TrainingDataWriter to write one game per chunk file
• Each game's training positions are now isolated in their own .gz file
• Removed batching logic that previously combined multiple games into one file

Testing

• Successfully built on Linux with GCC
• Verified output: Converting 5 games produces 5 separate files with varying sizes reflecting different game lengths

---

Made with Gemini and Claude

But dont merge just yet I still have to manually verify this code.