Ferrodoc

Ferrodoc is a pre-release, model-backed document extractor delivered as a Rust CLI plus native quality bundles for supported CPU/GPU substrates. The product goal is one-command output better than Marker without Python environments, containers, or manually operated model servers.

The Rust crates also preserve exact source evidence, geometry, model identity, and immutable document states for Foundry. Those are integration capabilities, not concepts normal CLI users need to manage.

Current status

The audited foundation provides bounded PDF processing, exact native geometry, append-only evidence and document states, engine isolation, model storage, deterministic caching, resource-aware planning, batches, renderers, and protected evaluation. The current bare-core/Tesseract path scores 36.4% on the complete public corpus. That is a control baseline, not the intended product. The first native LightOnOCR development bundle now runs end to end and produces strong reviewed output on the RP2040 probe, but it is not corpus-qualified, automatically distributed, or fast enough yet. Learned layout and specialist repair remain unqualified: Heron detection is integrated, but successful full-page OCR remains the default because an initial paired probe found proactive regional splitting could degrade semantic output. Regional table/formula/column OCR is retained as a bounded recovery path for resource-exhausted pages. The workspace stays at 0.1.0 until the owner chooses otherwise.

The current implementation sequence is PLAN.md, current truth is STATUS.md, and the competitive evidence is docs/marker-parity.md. CLI and protocol compatibility are documented in docs/cli.md and docs/protocol.md.

Prerequisites

• Rustup; the repository pins Rust 1.95.0 and requests Rustfmt and Clippy.
• Network access is needed only for the first Cargo dependency fetch. Builds, tests, and the smoke check do not download models or native binaries.

Verify the baseline

cargo metadata --locked --format-version 1 > /dev/null
./scripts/check-workspace.sh
./scripts/check-boundaries.sh
cargo fmt --all -- --check
cargo check --workspace --all-targets --locked
cargo test --workspace --locked
cargo clippy --workspace --all-targets --locked -- -D warnings
./scripts/smoke.sh
./scripts/benchmark-smoke.sh
./scripts/routing-smoke.sh
./scripts/readme-smoke.sh

Quick start

cargo run --locked -p ferrodoc -- --version
cargo run --locked -p ferrodoc -- fixtures/pdf/born-digital.pdf
cargo run --locked -p ferrodoc -- fixtures/pdf/born-digital.pdf --bundle /tmp/example.ferrodoc
cargo run --locked -p ferrodoc -- batch fixtures/pdf --output-dir /tmp/ferrodoc-batch --resume
cargo run --locked -p ferrodoc -- serve --bind 127.0.0.1:3210
cargo run --locked -p ferrodoc -- inspect fixtures/pdf/born-digital.pdf
cargo run --locked -p ferrodoc -- plan fixtures/pdf/born-digital.pdf
cargo run --locked -p ferrodoc -- convert fixtures/pdf/born-digital.pdf --output document.md
cargo run --locked -p ferrodoc -- explain fixtures/pdf/born-digital.pdf
cargo run --locked -p ferrodoc -- hardware
cargo run --locked -p ferrodoc -- plugins doctor
cargo run --locked -p ferrodoc -- models list --store .ferrodoc/models
cargo run --locked -p ferrodoc -- router inspect . benchmarks/routing/dataset.json

convert defaults to Markdown; --format html, --format json, and --format chunks select semantic HTML, the complete evidence graph, or flat retrieval records. --mode fast|quality|evidence makes the operating policy explicit, and --pages accepts zero-based indexes and ranges. See the CLI guide for bundle, batch, resume, sharding, and local HTTP contracts.

Ferrodoc never downloads models during build or conversion. Checked manifests record exact sizes, SHA-256 digests, source, revision, license, and acceptance requirements. models pull installs already acquired files atomically; see models/README.md. --model-dir DIR accepts the current LightOnOCR development bundle or the older OCRS pair. A LightOnOCR bundle also contains ferrodoc-lightonocr-worker (or the worker may be explicitly supplied with FERRODOC_LIGHTONOCR_WORKER during development); its file names select the engine automatically. This explicit local setup is not the final automatic installation experience.

plan accepts profiles plus hard --max-ram, --max-vram, --max-cost-microusd, and --deadline-ms constraints. Unknown values fail hard limits unless --allow-unknown-estimates explicitly requests guarded execution. --cache-dir DIR enables atomic stage caching from input, model, engine, schema, page, seed, and normalized-parameter identity.

Use --document-profile baseline or --mode evidence for the FP2 full-document baseline: every selected survey-nonblank page is rendered and OCRed even when native text is present. Native and OCR evidence remain separate.

The M-series deterministic table path reconstructs regular born-digital aligned tables with exact source spans and defensible geometry. The complete public table score rose from 0.2% to 29.4%, and the retained RP2040 pin table renders all ten data rows. This remains a narrow regular-table capability, not a scanned, merged-cell, or arbitrary-table claim; see FDX3 targeted tables and the current evaluation.

FP4 adds a runtime-owned durable storage-provider seam for immutable deltas, retained state manifests, and optional canonical DocumentIR checkpoints. Deterministic refinement reuse is pinned to the source, input state, producer/build, scope/config, schema, and seed; physical storage choices remain outside logical state identity. See FDX4 durable state reuse.

FP5 adds explainable hard-admitted local alternatives with deterministic plan identity, source-bearing fixed-point outcome/resource uncertainty, targeted versus whole-document escalations, and true Pareto retention. It deliberately leaves corpus priority and global utility to Foundry; see FDX5 explainable Pareto planning.

The deterministic foundry, real regression corpus, evaluator contracts, metrics, measurement evidence, held-out rules, and Pareto comparison workflow are described in Benchmarking and corpus governance. The default benchmark smoke is offline and explicitly verifies that missing work scores as failure rather than success.

The router and experiment commands are offline. router train writes a model only after re-hashing every conversion trace and benchmark report; its qualification field remains rejected unless it beats all declared deterministic baselines on identical held-out cases. research run reads immutable reports, re-hashes protected truth and evaluator files before and after scoring, observes cumulative budgets, and atomically writes resumable ledger state.

Optional Tesseract is selected explicitly and discovered at runtime; the default binary never links a native OCR library:

cargo run --locked -p ferrodoc --features tesseract -- plugins doctor --inference
cargo run --locked -p ferrodoc --features tesseract -- convert scan.pdf --ocr-engine tesseract

The experimental command wrapper requires a trusted FERRODOC_COMMAND_CONFIG, an absolute canonical executable allowlist, typed arguments, and process transport. It is not selected by the CLI as an official OCR engine. Fixed-corpus portfolio reports are produced by scripts/engine-qualification.sh and retain failures plus explicit unknown resources.

Design invariants

• Native PDF evidence is not overwritten by OCR evidence.
• Runtime-agnostic contracts do not depend on model, OCR, GPU, HTTP, or PDF runtimes.
• Unknown resource use is explicit rather than being reported as zero, and does not satisfy hard limits by default.
• The default born-digital path is offline-capable and CPU-capable.
• Deterministic expensive work is cacheable from complete semantic identity.
• Plugin stdout is reserved for framed protocol traffic.

Ferrodoc is dual-licensed under MIT or Apache-2.0.

Roadmap

M1-M8 are one product program rather than release gates. They turn the existing Rust/provenance substrate into a model-backed extractor with native plugin bundles, automatic hardware placement, clean output, and measured quality above Marker. See PLAN.md and STATUS.md.