PDF ins repo eat RAM (and RAM is expensive)

[NilsWildt]

Describe the bug

Indexing a directory containing a modest batch of small-to-medium PDFs causes gortex daemon's memory to grow unboundedly and never plateau. Isolated repro: a directory of 21 PDFs (40 KB–2.8 MB each, ~39 MB total, nothing over 3 MB) drove daemon RSS from a ~150 MB baseline to 1.2 GB → 2.7 GB in under 2 seconds, still climbing when I killed the process. In the original (non-isolated) run it reached 91.9 GiB before the OS killed it, and because the affected repo stayed in the tracked-repo list, every subsequent daemon restart re-triggered the same blowup during warmup — a crash-restart loop that never self-resolved.

To Reproduce

1. Point gortex at a directory with ~20 modest PDFs (1–3 MB each, no single file large):

   gortex track /path/to/repo-with-pdfs --wait
   

2. Watch the daemon process RSS (ps -o rss= -p $(cat ~/.gortex/cache/daemon.pid)) — it climbs into multiple GB within a couple of seconds instead of settling.

Isolation steps that narrowed it down:

• Same repo, restricted to just the PDF-heavy subdirectory (21 files, 39 MB) → RSS 1.2 GB → 2.7 GB in <2s, unbounded.

• A single PDF from that set (2.8 MB) tracked alone → RSS rose to ~350 MB for that one file (disproportionate for a 2.8 MB file, but it settles — not runaway by itself). So this isn't one poison-pill file; it's concurrent/cumulative PDF processing that never levels off.

• The same repo's 56 .jl files and other non-PDF content indexed in milliseconds with single-digit-MB memory, ruling out an unrelated cause in that repo.

• Workaround: excluding PDFs for the repo in ~/.gortex/config.yaml fixes it —

  repos:
    - path: /path/to/repo
      exclude:
        - "**/*.pdf"

  With that exclude the same repo indexes cleanly (251 files, 4,396 nodes, no RSS spike).

Expected behavior

Indexing a handful of MB-scale PDFs should cost a small, bounded multiple of their on-disk size — the way regular source files do, via the shared parse memory budget / shadow-buffer admission control (SetSharedParseMemoryBudget in internal/indexer/indexer.go). Not gigabytes of RSS with no plateau. At minimum, PDF extraction should fail closed (skip the file, log a warning) rather than take the whole daemon down.

Environment:

• OS: macOS 26.6.2 (Darwin 25.6.0), arm64, 24 GiB RAM
• Go version: go1.27.0 darwin/arm64
• Gortex version: v0.63.8-84-ged6af7c7+ed6af7c7 (commit ed6af7c7, built 2026-08-26T08:52:21Z)

Additional context

• PDFExtractor (internal/parser/languages/multimodal.go) already caps per-page text to 4000 chars and implements StreamingExtractor specifically so "a large document is never held whole in memory" per its own doc comment — so this doesn't look like an obvious whole-file-buffering bug in gortex's own wrapper.
• The likely site is pdfPageText → p.GetPlainText(nil) (from github.com/ledongthuc/pdf), called once per page via r.Page(page). My guess is either that library's content-stream/font decoding is pathologically expensive for a construct common to a subset of these PDFs, or many PDFs are being decoded concurrently by the parse worker pool with no per-type memory budget analogous to the one source-code parsing gets.
• The daemon's daemon.memory_limit (10 GiB in my config) is only a soft debug.SetMemoryLimit GC target — nothing hard-stops growth once GC can't keep up, which is how this got to 90+ GiB instead of failing fast.