PR: refactor: isolate website figure cleanup and publish latex-clean as built-in SDBS behavior - #18
Closed
metasmile wants to merge 43 commits into
Closed
PR: refactor: isolate website figure cleanup and publish latex-clean as built-in SDBS behavior#18metasmile wants to merge 43 commits into
metasmile wants to merge 43 commits into
Conversation
…h hook in build pipeline
…+ _publish cleanup patterns
- Fix: run_publish_sequence() is local to build.py, not sdb.publish - Add examples/publish-test/: end-to-end test for --website --publish - Verified: website build, LLMS sync, post-render, publish hook all succeed Refs: #15
….gitignore - Remove --publish implies --website (publish works independently) - Add /_publish/ to default and advanced template .gitignore - Add /_publish/ to project root .gitignore - Verified: sdb build --publish (standalone) succeeds Refs: #15
- run_publish_sequence() orchestrates per-target bundle creation
- _capture_tex(): quarto render --to latex -> _publish/{t}/source/
- _capture_md(): quarto render --to gfm -> _publish/{t}/{t}.md
- _copy_pdf(): locate existing PDF -> _publish/{t}/{t}.pdf
- _gen_metadata(): extract Quarto frontmatter + build.yml -> metadata.yaml
- Verified: sdb build --publish produces _publish/{index,paper}/ with correct artifacts
Refs: #15
- get_publish_dir() docs_root.parent/_publish → docs_root/_publish - _publish는 _site와 같은 디렉토리에 생성됨 - sdb clean에서 _site와 함께 _publish도 정리됨 - .gitignore 템플릿에 /_publish/ 이미 포함됨 Refs: #15
…fallback - _capture_tex: check _freeze/ and QMD dir for .tex before rendering - _capture_md: check _llms/*.llms.md and _site/*.llms.md first - _copy_pdf: check _site/ and QMD dir - Render --to latex / --to gfm only when pre-existing files not found Refs: #15
- .tex, .bib, figures go directly in _publish/{target}/
- {target}/ folder IS the publish product
Refs: #15
- _copy_source_context(): QMD + _include/ + _quarto.yml + bib + images preserving original directory structure - Generated artifacts (tex, pdf, md) placed alongside source files - Bundle is self-contained: paths work for re-render/compile Refs: #15
…mpilation)
- _capture_artifact_tex was deleting {stem}_files/ instead of copying
it to the bundle. Generated figures (code cells, plots) live in
_files/ and are referenced by \includegraphics{...} in the .tex.
- _capture_artifact_md gfm cleanup kept as-is (no _files/ references in .md)
Refs: #15
- build_targets() returned False on any failure before reaching the publish hook. Now publish runs for successful targets regardless. - Both website and non-website paths updated. Refs: #15
- _quarto.yml and _quarto-website.yml excluded (not publish artifacts) - Added _copy_artifact_html(): copies .html from _site/ or QMD dir Refs: #15
- Remove --to latex and --to gfm fallback renders from publish.py (Quarto already rendered everything during the build) - Remove dead _DEFAULT_POST_PUBLISH and run_publish_sequence() from build.py - Remove redundant publish hook in website success path - publish.py now only copies existing artifacts (html, pdf, md, tex, _files/) Refs: #15
- _capture_artifact_tex now also searches _site/{qmd_rel_dir}/{stem}_files/
for generated figures (SVG/HTML output from website build)
- Also checks _freeze/{stem}/{stem}_files/ and {qmd_dir}/{stem}_files/
- PDF-format figures (figure-pdf/) are ephemeral (LaTeX only);
SVG figures from _site/ are the best available fallback
Refs: #15
- _capture_files_for_publish() in build.py: called only when publish=True,
captures {stem}_files/ to both .staging/ and _cached/ right after render
- _capture_artifact_tex() in publish.py: checks .staging/ -> _cached/ -> _site/
in priority order, packaging all discoverable figures
- _include/ removed from bundle (not referenced by .tex/.html/.md)
- Works on both cache hits and fresh renders
Refs: #15
…ports, CLI help - publish.py: add missing List, Optional imports (RuntimeError fix) - publish.py _capture_artifact_tex(): add .staging/ and _cached/ lookup - build.py _capture_files_for_publish(): remove redundant self-import - cli.py: fix --publish help text (not 'Implies --website') Refs: #15
…d/ for figures - Previous commit claimed this change but the replacement didn't match (escaping issue). Now properly implemented. - Sources: .staging/ -> _cached/ -> QMD dir -> _freeze/ -> _site/ - Deduplication by resolved path to avoid redundant copytree Refs: #15
…iles/...}
- _copy_source_context: copy _files/ from QMD parent dir (referenced by .tex)
- .tex compilation now resolves \input{_files/whitepaper_metadata.tex}
- figure-pdf/ and mediabag/ remain missing due to Quarto latex-clean timing
(cannot be captured from outside Quarto's render process)
Refs: #15
…tex compilation) Refs: #15
- Previously iterated all cached targets * all hashes even after finding
the matching {stem}_files/. Now breaks both inner and outer loop.
- Fixes performance issue on repos with many targets (e.g. SSCCS: 65)
Refs: #15
Covers: - get_publish_dir, _site_path path helpers - _copy_source_context (bib, images, _files/ metadata) - _capture_artifact_tex (from QMD dir, from staging, missing) - _capture_artifact_md (from _llms/, _site/, missing) - _copy_artifact_pdf / _copy_artifact_html (site root, site subdir, QMD dir) - _gen_metadata (frontmatter extraction, fallback, config merge) - run_publish_sequence (no targets, unknown target, bundle assembly) - _capture_files_for_publish (staging, cache, no files) - CLI --publish / --zenodo flag forwarding Refs: #15
…emoved) - Remove --zenodo, --zenodo-sandbox CLI flags - Remove zenodo params from build_targets() - Remove zenodo block from publish sequence - Rename config key: publish.zenodo.metadata -> publish.metadata - metadata.yaml format stays (generic metadata, not zenodo-specific) - All tests pass (192/192)
- _publish/ is deleted before each build, same as _site/ - Both are cleaned by sdb clean and listed in .gitignore Refs: #15
- test_publish_removes_existing_publish_dir - test_no_publish_does_not_clean_publish - test_site_always_cleaned_regardless_of_publish Total: 195 tests pass
…r future use) - Remove call to _capture_artifact_md from run_publish_sequence - Keep the function itself (for potential website-side use) - Update bundle test to not expect .md
…render - Move publish from before failure check to after post_render - Add early publish before failure check for partial failures - _capture_artifact_md now finds .llms.md from _llms/ (synced) - Verified: MD from _llms (not _site)
…/ content from being scanned as build targets)
- test_clean_publish_before_init: _publish/ cleaned when --publish flag set - test_no_clean_without_publish_flag: _publish/ untouched without flag Total: 197 tests pass
…t figure pollution)
- _cached/{bundle.name}/ only, not _cached/*/*/
- bundle.name == target name = publish dir basename
- Zero risk of picking up figures from other targets' caches
… test)
- test_cached_scope_respects_target_boundary: verifies _capture_artifact_tex
does not pick up {stem}_files/ from other targets' caches
- Total: 198 tests pass
PR: feat: add --publish mode for reproducible artifact bundle generation
- Prevents _publish/ content from being discovered as build targets - Matches existing _llms/ exclusion pattern - Safety net: _publish/ cleanup in cli.py remains for existing projects
When building with --publish, automatically: - Set website mode if not already enabled - Disable latex-clean to preserve .tex and figure-pdf files - Capture .tex files alongside _files/ for both staging and caching - Include site_libs in HTML bundles - Update .staging path for artifact discovery Also update workflow tests to verify publish outputs including figure-pdf, site_libs, and .tex compilation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors figure cleanup and latex-clean into SDBS built-in behavior, decoupled from
_quarto-website.ymlconfiguration. The published_quarto-website.ymltemplates no longer carrypost-renderdirectives.Changes
src/sdb/build.py_clean_website_figures(): deletesfigure-pdf,mediabag,figure-beamerfrom_site/after website build. Only runs in--websitemode (not--publish). Injectlatex-clean: falsefor--publishmode (preserves.tex).src/sdb/templates/*/_quarto-website.ymlpost-render(figure cleanup is now built-in Python)tests/test_publish.pyDesign
_clean_website_figures():os.walkover_site/, removesfigure-pdf/,mediabag/,figure-beamer/directorieslatex-clean: false: injected into_quarto-website.ymlbefore publish build, restored infinallyblock_quarto-website.yml— no configuration side effectsRefs: #15