Skip to content

feat: support standalone HTML documents - #53

Open
xianjianlf2 wants to merge 4 commits into
firecrawl:mainfrom
xianjianlf2:feat/standalone-html
Open

feat: support standalone HTML documents#53
xianjianlf2 wants to merge 4 commits into
firecrawl:mainfrom
xianjianlf2:feat/standalone-html

Conversation

@xianjianlf2

@xianjianlf2 xianjianlf2 commented Aug 7, 2026 •
edited by blacksmith-sh Bot
Loading

Copy link
Copy Markdown

Summary

Closes #52.

  • Add standalone HTML detection from <!doctype html> or an <html> root, including HTML exports saved with a .doc extension.
  • Reuse the EPUB HTML-to-model walker for headings, formatting, links, tables, lists, CSS visibility, and external images.
  • Treat HTML void elements such as unclosed <meta> and <img> as empty elements in the shared DOM parser.
  • Expose html / .html / .htm through the Rust, Node, Python, and WASM APIs and update docs/tests.
  • MHTML remains out of scope for this change.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --locked
  • npm test
  • python -m unittest discover -s python/tests
  • node --test wasm/test.mjs

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@xianjianlf2

xianjianlf2 commented Aug 7, 2026

Copy link
Copy Markdown
Author

Follow-up: synchronized node/package-lock.json so the CI npm ci step now succeeds.

Local validation after the follow-up commit:

  • npm ci --ignore-scripts
  • npm test (13/13)
  • cargo test --locked
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • Python and WASM binding tests

The upstream CI run is currently action_required, which requires a maintainer of firecrawl/anydoc to approve the fork workflow.

@xianjianlf2

xianjianlf2 commented Aug 11, 2026

Copy link
Copy Markdown
Author

Merged current origin/main into feat/standalone-html and resolved the node/package.json release conflict by keeping version 0.1.8 while preserving the HTML support description.

Validation after merge:

  • npm ci --ignore-scripts
  • npm test (13/13)
  • cargo test --locked (212 unit + robustness + snapshots passed; 1 samples test ignored as expected)
  • cargo clippy --workspace --all-targets --all-features -- -D warnings

@xianjianlf2

xianjianlf2 commented Aug 11, 2026

Copy link
Copy Markdown
Author

Rechecked current head c8a1b73; no code update was needed.

Local validation:

  • cargo test --locked (unit/robustness/snapshots passed; samples_sweep remains ignored)
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • npm --prefix node ci --ignore-scripts
  • npm --prefix node run build
  • npm --prefix node test (13 passed)
  • git diff --check

Note: npm test fails before npm run build because the local native binding artifact is not present; after the repo's build step it passes. GitHub currently shows cubic success and mergeStateStatus=UNSTABLE.

@itsaaronngan itsaaronngan mentioned this pull request Aug 27, 2026
Open
@xianjianlf2
Merge remote-tracking branch 'upstream/main' into feat/standalone-html
b43c39b 
# Conflicts:
#	node/package.json
#	node/test.mjs
#	python/tests/test_anydoc.py
#	src/lib.rs
#	src/package/xml.rs
#	wasm/test.mjs
@xianjianlf2

xianjianlf2 commented Aug 29, 2026

Copy link
Copy Markdown
Author

Resolved conflicts against upstream main.

Validation:

  • cargo test (290 unit tests passed, robustness passed, snapshots passed with 1 ignored sample sweep)
  • cargo fmt --check
  • npm run build --prefix node && npm test --prefix node (19 passed)
  • uv venv .venv && . .venv/bin/activate && uv pip install maturin && maturin develop --manifest-path python/Cargo.toml && python -m unittest discover -s python/tests (11 passed)
  • wasm-pack build wasm --release --target web && node --test wasm/test.mjs (8 passed)
  • git diff --check

Current head: b43c39b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feature request: Support for HTML / MHTML files (e.g. Jira exports)

1 participant

@xianjianlf2