Skip to content

feat: add standalone HTML support - #147

Open
marcellmanfrin wants to merge 27 commits into
firecrawl:mainfrom
marcellmanfrin:feature/html-support-pr
Open

feat: add standalone HTML support#147
marcellmanfrin wants to merge 27 commits into
firecrawl:mainfrom
marcellmanfrin:feature/html-support-pr

Conversation

@marcellmanfrin

@marcellmanfrin marcellmanfrin commented Aug 28, 2026

Copy link
Copy Markdown

Scope

Adds standalone HTML support (.html / .htm) for the HTML portion of #52. MHTML/MHT remains out of scope here and is handled separately in #149.

Summary

  • adds Format::Html, extension mapping, UTF-8/UTF-16 content detection, charset sniffing, and tolerant HTML5 parsing via scraper/html5ever
  • reuses the shared semantic HTML conversion path used by EPUB
  • preserves CSS semantics, links, anchors, and non-empty absolute/protocol-relative/relative image references without fetching local or remote resources
  • enforces pre-DOM node/depth limits plus post-parse defense-in-depth checks
  • exposes HTML through Rust, Node, Python, WASM and CLI bindings
  • includes controlled fixtures plus real LibreOffice DOCX/XLSX HTML exports and snapshot coverage

Review follow-ups

The current head includes the earlier charset, detector-precedence, malformed-HTML repair, list, heading, foreign-content, and anchor-preflight fixes.

The latest Cubic findings discovered while reviewing #149 were reproduced against the previous #147 head and fixed here because they belong to the shared HTML frontend:

  • valid HTML5 frameset documents no longer fail solely because they have no <body>; they convert to an empty document when there is no renderable body content
  • non-empty relative <img src> values are preserved as image references, just like absolute and protocol-relative references; anydoc still does not fetch or load those resources
  • the LibreOffice corpus snapshot now intentionally preserves the two relative image references that were previously dropped

TDD evidence

RED branch: audit/html-cubic-frameset-relative-red, based on previous head c6b7bb18608d4bde426a71d9a5300070d8f653fc.

GitHub Actions run 33300432395, job 99227423622 reproduced both findings:

  • frameset conversion returned Malformed("HTML parser produced no body element")
  • the relative-image document produced no block because the image source was discarded

Validated functional SHA: ce7948287c8b4a62666f851a5845beabb091ef38.

The functional diff from the previous head is limited to src/formats/html.rs, tests/html.rs, and the expected LibreOffice snapshot update.

Fresh full validation

Temporary CI-only branch: verify/html-cubic-frameset-relative-full-v2.

CI-only commit: 56cf2e57fc02f17c20f1ede780524b99e4acaed2. Its sole parent is the validated functional SHA ce7948287c8b4a62666f851a5845beabb091ef38; every job explicitly checks out that functional SHA detached.

Authoritative GitHub Actions run: 33300809474 — all 6 jobs passed.

  • latest frameset and relative-image regressions: passed
  • full html, html_list, html_corpus, snapshots, and content-detection regressions: passed
  • real HTML root extracted from the supplied SEI/TRE-PB MHTML fixture: passed
  • Rust fmt, workspace/all-targets/all-features Clippy with -D warnings, and cargo test --locked: passed
  • Rust MSRV 1.88 check and HTML test compilation: passed
  • Node npm ci, build, tests, and committed binding determinism: passed
  • WASM wasm32 Clippy, release build, and node --test wasm/test.mjs: passed
  • Python locked Maturin release wheel, installed compiled site-packages module, and full unittest suite: passed

Real validation fixture:

  • MHTML input: 505,883 bytes; SHA-256 387c6f2d7223da224a8f55962b97eac947734c97fc84888e1e5619e93745837c
  • extracted HTML root: 200,381 bytes; SHA-256 32fca03d52dc62518bf4419e2a4b1be96160195cd99f613a2d3230f1bb1e817c
  • standalone HTML Markdown: 78,154 bytes; SHA-256 6119ae4bb085acabe21149492020e159e431de4f7a169350b1b619f2add61c1a
  • expected Portuguese content preserved; no , Ã, or  mojibake markers

The current PR head is exactly the validated functional SHA above.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 16 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread src/formats/html.rs Outdated
Comment thread src/formats/detect.rs Outdated
Comment thread src/formats/detect.rs
Comment thread src/formats/detect.rs Outdated
Comment thread src/formats/detect.rs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/formats/html.rs Outdated
Comment thread src/formats/html.rs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/formats/html.rs">

<violation number="1" location="src/formats/html.rs:179">
P1: For HTML-style `<div/>` chains, this condition undercounts nesting because non-void self-closing flags are ignored by the HTML tree builder. Track those elements as open, while handling foreign-content self-closing elements separately, so pathological depth is rejected before DOM construction.</violation>

<violation number="2" location="src/formats/html.rs:222">
P2: When malformed HTML relies on HTML5 heading repair, the preflight rejects it as too deep even though html5ever would produce sibling headings. Add the heading implied-closing rule (and the other HTML5 tree-builder closures) before applying the depth limit.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/formats/html.rs Outdated
Comment thread src/formats/html.rs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/formats/detect.rs">

<violation number="1" location="src/formats/detect.rs:76">
P2: A UTF-16 doctype with more than 51 whitespace code units between `DOCTYPE` and `html` is not detected because the fixed prefix ends before the name. Scan the doctype marker state instead of truncating detection at 64 code units.</violation>
</file>

<file name="Cargo.toml">

<violation number="1" location="Cargo.toml:38">
P3: This change drops the trailing newline at the end of Cargo.toml (`strip = "symbols"` now ends the file with no newline). The target branch ends the file with a trailing newline. Restore the trailing newline to keep the file clean and avoid noisy diffs.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/formats/detect.rs Outdated
Comment thread Cargo.toml Outdated
marcellmanfrin added a commit to marcellmanfrin/anydoc that referenced this pull request Aug 29, 2026
marcellmanfrin added a commit to marcellmanfrin/anydoc that referenced this pull request Aug 29, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 16 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/shared/html.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant