DocMark Bench

DocMark Bench is an end-to-end Streamlit workbench for generating Markdown with several open-source document parsers, viewing the original beside every result, and ranking each conversion route for the current document.

It integrates:

• MarkItDown
• LiteParse
• PyMuPDF4LLM
• Kreuzberg v4 LTS
• PdfItDown
• tiktoken for exact output token counts

What it does

• Accepts local documents or public HTTP(S) document/webpage URLs.
• Supports PDF, DOCX, XLSX, PPTX, HTML, CSV, JSON, XML, Markdown, text, common images, OpenDocument, legacy Office, email, and EPUB inputs at the ingestion layer.
• Runs converters in separate killable subprocesses, sequentially by default for less CPU-contended timing.
• Keeps one engine's exception, native crash, missing dependency, or timeout from failing the whole benchmark.
• Shows the original document next to dynamic tabs containing every converter result.
• Reports end-to-end route time, UTF-8 Markdown size, tiktoken count, package versions, route type, and errors.
• Calculates document-specific quality, speed, compactness, and overall scores.
• Accepts optional reference Markdown for stronger similarity-based quality evaluation.
• Downloads individual Markdown, a benchmark CSV, or a ZIP containing all successful outputs and a JSON manifest.
• Retains only the current result in per-user Streamlit session state; temporary worker files are automatically deleted.

Important PdfItDown distinction

The current PdfItDown package converts inputs to PDF; it is not a standalone Markdown extractor. The app therefore labels its row as:

PdfItDown → PyMuPDF4LLM

That composite result measures PdfItDown normalization and PyMuPDF4LLM extraction together. It is never presented as pure PdfItDown Markdown quality. PdfItDown is also used as an optional, explicitly labeled normalizer for PDF-only routes.

Architecture

flowchart LR
    A["Upload or public URL"] --> B["Input, size, signature, and SSRF guards"]
    B --> C["Temporary per-run workspace"]
    C --> D["Sequential benchmark orchestrator"]
    D --> E1["MarkItDown worker"]
    D --> E2["LiteParse worker"]
    D --> E3["PyMuPDF4LLM worker"]
    D --> E4["Kreuzberg worker"]
    D --> E5["PdfItDown + PyMuPDF4LLM worker"]
    E1 & E2 & E3 & E4 & E5 --> F["Metrics and explainable scoring"]
    F --> G["Side-by-side Streamlit workspace"]
    F --> H["Markdown, CSV, ZIP, and JSON downloads"]

Loading

The Streamlit entry point contains UI code only. Input validation, safe previews, capability routing, workers, scoring, and report construction live in independently testable modules under src/docmark_bench/.

Capability routing

No upstream package supports every accepted input natively. The application records how every result was produced:

Engine row	Direct formats in this app	Optional normalization
MarkItDown	PDF, DOCX, XLS/XLSX, PPTX, HTML, text/markup, common images, email, EPUB	None
LiteParse	PDF, DOCX, XLSX, PPTX, common raster images	PdfItDown for its other PdfItDown-compatible inputs; Office inputs also normalize when LibreOffice is absent
PyMuPDF4LLM	PDF in the fully open-source configuration	PdfItDown for Office, HTML/markup, text, and common images
Kreuzberg	Broad direct surface including PDF, Office, OpenDocument, web/markup, email, EPUB, and images	None
PdfItDown → PyMuPDF4LLM	Not a direct extractor	Always a composite PDF-normalization route

Unsupported combinations remain visible with an unsupported status. They are not assigned a score or silently replaced with another parser.

PyMuPDF4LLM's optional Office support requires PyMuPDF Pro, so this application intentionally uses its open-source PDF route and labels any PdfItDown-normalized input.

Benchmark methodology

For every successful output:

• Processing time is measured inside the worker around the whole conversion route. Normalized and composite timings include PdfItDown.
• Output size is the byte length of UTF-8 Markdown.
• Tokens are counted with the selected tiktoken encoding (o200k_base by default).
• Speed score is relative to the fastest successful route for the current document.
• Compactness score compares token count with the successful-output median.

Without reference Markdown, quality is an explicitly labeled proxy:

quality = 35% coverage + 35% peer consensus + 20% character health + 10% Markdown structure
overall = 70% quality + 20% speed + 10% compactness

With reference Markdown:

quality = 80% reference similarity + 15% character health + 5% Markdown structure
overall = 70% quality + 20% speed + 10% compactness

The winner is therefore the best route for the submitted document and selected scoring mode, not a universal package ranking. For a serious package evaluation, run a representative corpus with curated reference Markdown and aggregate the exported CSVs.

Local development

Prerequisites:

• Python 3.12
• uv
• LibreOffice on PATH for LiteParse's direct DOCX/XLSX/PPTX conversion. If it is unavailable, the app can use its labeled PdfItDown fallback.

Install the locked environment and run:

uv sync --extra dev
uv run streamlit run streamlit_app.py

Open http://localhost:8501.

Quality checks:

uv run ruff check .
uv run mypy src/docmark_bench
uv run pytest --cov=docmark_bench

Docker deployment

The production image runs as a non-root user and installs LibreOffice, Tesseract, fonts, and libmagic for document handling:

docker build -t docmark-bench .
docker run --rm -p 8501:8501 --memory=4g --cpus=2 docmark-bench

The image exposes /_stcore/health through a container health check. In a multi-instance deployment, use sticky sessions because Streamlit session state is process-local.

For large or untrusted public workloads, run converter workers in a dedicated sandbox/queue tier with OS-level CPU and memory quotas. The included subprocess isolation handles exceptions and timeouts on one host, but it is not a substitute for a hardened multi-tenant container sandbox.

Runtime configuration

Environment variables are bounded by safe application limits:

Variable	Default	Purpose
DOCMARK_MAX_UPLOAD_MIB	50	Local upload cap
DOCMARK_MAX_DOWNLOAD_MIB	50	Remote decompressed download cap
DOCMARK_MAX_OUTPUT_MIB	10	Per-engine Markdown cap
DOCMARK_CONVERTER_TIMEOUT_SECONDS	120	Default per-engine timeout
DOCMARK_URL_CONNECT_TIMEOUT_SECONDS	5	Remote connect timeout
DOCMARK_URL_READ_TIMEOUT_SECONDS	20	Remote read timeout
DOCMARK_MAX_REDIRECTS	3	Maximum validated redirects

See .env.example.

Security and resilience controls

• File extensions are allow-listed and PDF/ZIP-based document signatures are checked.
• Filenames are reduced to safe basenames before writing into a temporary workspace.
• Upload, decompressed remote-download, expanded Office archive, generated output, page, and timeout limits are enforced.
• Remote URLs allow only HTTP(S), standard ports, no embedded credentials, and public IP resolutions. Redirect targets are revalidated.
• Internet-facing deployments should additionally enforce an outbound firewall or egress proxy; application-level SSRF validation is defense in depth and cannot replace network policy.
• Uploaded HTML is allow-list sanitized and rendered with JavaScript disabled. It is never passed to same-origin st.iframe.
• Office previews reject excessive entry counts, expanded sizes, and suspicious compression ratios.
• Image previews use Pillow's decompression-bomb protection and are re-encoded before display.
• Workers receive a small allow-list of environment variables instead of the entire server environment.
• Worker process trees are terminated on timeout.
• MarkItDown plugins and network/cloud converters are disabled.
• Kreuzberg cache is disabled for timing fairness and to avoid cross-user document persistence.
• The Streamlit server hides detailed framework tracebacks from clients and enables XSRF protection.
• CI runs linting, strict type checks for application modules, tests, and coverage.

Operational limitations

• Conversion quality varies substantially with scans, handwriting, dense tables, charts, multi-column layouts, fonts, and damaged documents.
• OCR increases latency and may require language data that is not installed in a minimal deployment.
• URL pages that require authentication, client-side JavaScript, non-standard ports, or private network access are intentionally unsupported.
• The application does not execute browser JavaScript to capture dynamic webpages; it downloads the server response.
• Direct Office fidelity depends on upstream libraries and LibreOffice availability.
• The five engines have different licenses. In particular, review the AGPL/commercial licensing terms of PyMuPDF and PyMuPDF4LLM before distributing or offering this application as a network service. This repository does not provide legal advice.

Project layout

.
├── streamlit_app.py              # Streamlit UI
├── src/docmark_bench/
│   ├── ingestion.py              # Upload and SSRF-safe URL ingestion
│   ├── preview.py                # Sanitized, bounded native previews
│   ├── registry.py               # Capability matrix and route labels
│   ├── worker.py                 # One-engine subprocess entry point
│   ├── service.py                # Timeout and fault-isolated orchestration
│   ├── metrics.py                # tiktoken and quality scoring
│   └── reporting.py              # CSV, ZIP, and JSON exports
├── tests/                        # Unit, integration, and Streamlit smoke tests
├── .streamlit/config.toml        # Server hardening and Fluent theme
├── Dockerfile                    # Non-root production image
├── pyproject.toml
└── uv.lock

Test Coverage