Journal-based linked chains, timestamp indexes, U128/result codes, module split, CI and repo hygiene

.github/workflows/tb_ocaml_ci.yml

@@ -27,29 +27,28 @@ jobs:
       - name: Set up OxCaml
         uses: ocaml/setup-ocaml@v3
         with:
-          ocaml-compiler: oxcaml-compiler.5.2.0minus31
+          ocaml-compiler: oxcaml-compiler.5.2.0minus39
+          dune-cache: true
           opam-repositories: |
             ox: git+https://github.com/oxcaml/opam-repository.git
             default: git+https://github.com/ocaml/opam-repository.git
 
       - name: Install dependencies
         run: |
-          opam install . --deps-only --with-test --with-doc
-          opam install -y ocamlformat.0.26.2+ox1
+          opam install . --deps-only --with-test
+          opam install -y ocamlformat.0.26.2+ox2
+
+      - name: Lint opam manifest
+        run: opam lint tigerbeetle_ocaml.opam
 
       - name: Build
         run: opam exec -- dune build
 
-      - name: Build docs
-        run: opam exec -- dune build @doc
-
       - name: Run Dune runtest
         run: opam exec -- dune runtest
 
       - name: Build benchmark
         run: opam exec -- dune build @bench
 
       - name: Check formatting
-        run: |
-          find . -name _build -prune -o \( -name '*.ml' -o -name '*.mli' \) -print0 \
-            | xargs -0 opam exec -- ocamlformat --check
+        run: opam exec -- dune build @fmt

.github/workflows/tb_ocaml_coverage.yml

@@ -24,18 +24,19 @@ jobs:
         with:
           submodules: recursive
 
-      - name: Set up OxCaml
+      # bisect_ppx and odoc do not currently build on OxCaml's opam overlay, so
+      # coverage and docs run on the upstream compiler; the core is Stdlib-only.
+      - name: Set up OCaml
         uses: ocaml/setup-ocaml@v3
         with:
-          ocaml-compiler: oxcaml-compiler.5.2.0minus31
-          opam-repositories: |
-            ox: git+https://github.com/oxcaml/opam-repository.git
-            default: git+https://github.com/ocaml/opam-repository.git
+          ocaml-compiler: "5.2"
+          dune-cache: true
 
       - name: Install dependencies
-        run: |
-          opam install . --deps-only --with-test
-          opam install bisect_ppx
+        run: opam install -y qcheck bisect_ppx odoc
+
+      - name: Build docs
+        run: opam exec -- dune build @doc
 
       - name: Run coverage-instrumented tests
         run: |
@@ -44,8 +45,21 @@ jobs:
           BISECT_SILENT=YES BISECT_FILE="$PWD/_coverage/bisect" \
             opam exec -- dune runtest --instrument-with bisect_ppx --force
 
-      - name: Generate coverage summary
-        run: opam exec -- bisect-ppx-report summary --coverage-path _coverage
+      - name: Enforce coverage threshold
+        env:
+          MINIMUM_COVERAGE: "75"
+        run: |
+          summary="$(opam exec -- bisect-ppx-report summary --coverage-path _coverage)"
+          echo "$summary"
+          percent="$(printf '%s\n' "$summary" | sed -n 's/.*(\([0-9.]*\)%).*/\1/p' | head -n1)"
+          if [ -z "$percent" ]; then
+            echo "could not parse coverage percentage" >&2
+            exit 1
+          fi
+          if awk -v p="$percent" -v m="$MINIMUM_COVERAGE" 'BEGIN { exit !(p + 0 < m + 0) }'; then
+            echo "coverage ${percent}% is below the ${MINIMUM_COVERAGE}% minimum" >&2
+            exit 1
+          fi
 
       - name: Generate HTML report
         run: opam exec -- bisect-ppx-report html --coverage-path _coverage

BENCHMARK_COMPARISON.md

@@ -1,9 +1,20 @@
 # State-machine benchmark status
 
-The OCaml workload creates two accounts before timing, then applies 30,000
-successful posted transfers in prebuilt batches of 30. Request construction is
-outside the timed interval. A future valid native runner should use the same
-transfer workload.
+The OCaml benchmark runs several workloads over 100 accounts, each with
+prebuilt requests in batches of 30 so request construction is outside the
+timed interval:
+
+| Workload | Timed path |
+| --- | --- |
+| `posted_transfers` | 30,000 successful single-phase transfers. |
+| `pending_then_post_or_void` | 30,000 pending transfers, then 30,000 posts/voids resolving them. |
+| `linked_chains` | 30,000 transfers in successful 30-request linked chains, over a pre-populated ledger. |
+| `failing_linked_chains` | 30,000 transfers in linked chains whose last request fails, exercising rollback. |
+| `queries_over_populated_ledger` | Timestamp-bounded `query_*`, `get_account_transfers`, and `get_account_balances` over 30,000 transfers. |
+| `pending_expiry` | Expiring 30,000 timed-out pending transfers. |
+
+A future valid native runner should use the `posted_transfers` workload for
+the paired comparison.
 
 Only the OCaml runner is currently executable:
 
@@ -38,4 +49,4 @@ modes.
 | Implementation | Operations/s | Mean batch latency (ms) | Allocation |
 | --- | ---: | ---: | --- |
 | TigerBeetle Zig | blocked | blocked | The standalone fixture needs TigerBeetle's internal commit sequencing completed before it can produce a valid run. |
-| OCaml | 1,250,115 | 0.024 | 5,767,131 words total; 192.24 words/op |
+| OCaml (`posted_transfers`, before indexing/journal work) | 1,250,115 | 0.024 | 192.24 words/op |

CONTRIBUTING.md

@@ -0,0 +1,86 @@
+# Contributing
+
+Read [`README.md`](README.md) and [`AGENTS.md`](AGENTS.md) first. The pinned
+TigerBeetle tree at `path/to/tigerbeetle` is the behavior oracle; do not edit
+or advance it as part of a change to the OCaml core.
+
+## Toolchain
+
+The OCaml code is built with the OxCaml compiler pinned in
+`.github/workflows/tb_ocaml_ci.yml`. From `ocam/`:
+
+```sh
+opam switch create tigerbeetle-oxcaml oxcaml-compiler.5.2.0minus39 \
+  --repos ox=git+https://github.com/oxcaml/opam-repository.git,default
+eval "$(opam env --switch tigerbeetle-oxcaml)"
+opam install . --deps-only --with-test
+opam install ocamlformat.0.26.2+ox2
+```
+
+## Checks
+
+Run the same commands CI runs before opening a pull request, from `ocam/`:
+
+```sh
+opam exec -- dune build
+opam exec -- dune runtest
+opam exec -- dune build @bench
+opam exec -- dune build @fmt     # or `dune fmt` to rewrite in place
+```
+
+`dune build @doc` needs `odoc`, which (like `bisect_ppx`) does not currently
+build on the OxCaml opam overlay; CI builds docs on upstream OCaml 5.2 in the
+coverage workflow.
+
+The coverage workflow instruments the tests with Bisect PPX and fails when
+line coverage drops below the `MINIMUM_COVERAGE` set in
+`.github/workflows/tb_ocaml_coverage.yml`. That workflow runs on upstream
+OCaml 5.2 (the core is Stdlib-only). Reproduce it locally on a standard switch
+with:
+
+```sh
+opam install bisect_ppx
+BISECT_FILE="$PWD/_coverage/bisect" \
+  opam exec -- dune runtest --instrument-with bisect_ppx --force
+opam exec -- bisect-ppx-report summary --coverage-path _coverage
+```
+
+## Layout of the OCaml core
+
+| Module | Role |
+| --- | --- |
+| `U128` | Unsigned 128-bit integers with explicit overflow/underflow results. |
+| `Types` | Account, transfer, filter, and status records shared by every module. |
+| `Result_code` | Numeric `CreateAccountsResult`/`CreateTransfersResult` codes. |
+| `Timeline` | Append-only, timestamp-ordered index with binary-searched range reads. |
+| `Ledger` | Storage, timestamp and per-account indexes, and the rollback journal. |
+| `State_machine` | Validation, batch execution, and the public API. |
+
+Keep the core deterministic and synchronous: no Async, storage, clock, or
+network dependency. New behavior should be compared against the pinned Zig
+`src/state_machine.zig` and its tests, and covered by a scenario in
+`ocam/test/state_machine_test.ml` or a property in
+`ocam/test/state_machine_property_test.ml`.
+
+## Version control
+
+Contributors use [Jujutsu (`jj`)](https://github.com/jj-vcs/jj) on top of the
+Git repository. Set it up once in an existing clone:
+
+```sh
+jj git init --colocate
+jj git fetch
+```
+
+With a colocated repository, Git and `jj` share the working copy, so CI and
+GitHub continue to see ordinary Git branches. Inspect `jj status` and
+`jj diff` before and after changes, and see the "Commit and push workflow" in
+[`AGENTS.md`](AGENTS.md) for how bookmarks are moved and pushed. Plain Git
+commands also work if you do not use `jj`; the requirement is that the history
+you push consists of coherent commits that do not touch the pinned submodule.
+
+## License
+
+Contributions are accepted under the Apache License 2.0 in [`LICENSE`](LICENSE),
+the same license as the upstream TigerBeetle sources this repository derives
+from.