Add directory tree UI (#156) and fix directory-filter performance#231
Merged
Conversation
…stant Move the hardcoded initial-load depth cutoff out of get-directory-tree.ts into its own dependency-free module and publish it via a browser-safe sub-export, so the viewer frontend's default-expanded-by-depth policy reads the same value instead of duplicating it as an independent literal.
Implement the frontend for issue #156's directory tree view — the backend read model and API landed in #107 with no frontend surface yet. Renders the initial depth <= 3 payload as an already-expanded tree per root host, expands boundary nodes on demand via /api/directory-tree/children (no hardcoded depth cutoff — driven purely by whether a node's children are already known), and shows a selected directory's direct pages in a paginated side panel reusing the existing DataTable virtual mode. Adds a dedicated E2E fixture that builds the viewer read model explicitly (the shared fixture doesn't), so directory-tree.spec.ts runs against real data without touching the other six views' fast-path/legacy dispatch in the shared-fixture suite.
…odel Add direct_html_page_count/descendant_html_page_count alongside the existing directPageCount/descendantPageCount columns, classified via classifyContentType at build time. The unfiltered counts include every crawled resource under a directory (images, PDFs, etc.), which is misleading when the viewer's tree UI labels the number "pages" — the new columns give callers an accurate HTML-only count without an extra query. Bumps VIEWER_READ_MODEL_SCHEMA_VERSION to 17.
…s to /pages Drop the in-place direct-pages panel (DirectoryPagesPanel, useDirectoryTreePagesInfinite) in favor of navigating to /pages?directory= <path>&contentTypeCategory=html on node select. The Pages view's directory filter already covers the whole subtree with full sort/filter/pagination, and contentTypeCategory=html keeps the destination to actual pages, not every crawled resource under that path (images, PDFs, etc.) — wires a one-line directory param read into pages-view.tsx to support it. Redesigns the tree's visual style: page-level scroll instead of a boxed pane, badges pulled closer to node names as a pill (no more far-apart justify-content: space-between), and indent guide lines replacing the even/odd row shading that would have visibly shifted on every expand/ collapse. Badges now read descendantHtmlPageCount instead of the unfiltered descendantPageCount, matching the query-layer change.
Show a folder icon (open/closed based on expanded state) before the directory name, move the expand/collapse control after the name and page count instead of leading the row, and render it as an SVG chevron (larger, rotates on expand) instead of a plain text glyph. A host's root row no longer shows the toggle at all — it is always expanded, since collapsing the one row every other row nests under would hide the whole tree behind a single click.
path_sort_key was already stored for exactly this range-scan use case but had no index and no reader. Add the vp_path index and a directory option to listViewerPages/applyViewerPagesFilters — a >=/< range on path_sort_key bounded by a maximal-codepoint sentinel, not a LIKE, so it matches the whole subtree while still seeking the index regardless of collation settings. Bump VIEWER_READ_MODEL_SCHEMA_VERSION to 18 for the new index; existing archives fall back to the legacy path until their read model rebuilds. Measured against a 451k-page archive: a directory-filtered /api/pages request that previously did not return within 2 minutes (the legacy path's substring LIKE against url_refs.url can't seek an index) now completes in under 100ms.
…ory tree Add a depth input + button that resets every node back to the plain depth < N default, discarding expand/collapse overrides entirely rather than computing which individual nodes to flip. Add a sort-order select + button (path / most pages first / fewest pages first) that reorders siblings at every level by descendantHtmlPageCount, computed client-side with no extra request. Move all of this view's state (expand/collapse overrides, collapse depth, sort order) into the URL via useUrlFilter, matching every other list view's convention. Selecting a node navigates away to the Pages view, which unmounts this component — a plain useState reset to its initial value on the back button, discarding whatever the user had expanded or configured. Format the page-count badge with toLocaleString() for thousands separators, matching the convention already used elsewhere in this view layer.
Rebuilds the viewer read model for archives predating viewer_directory_nodes.direct_html_page_count / descendant_html_page_count (schema version < 17). No write-model change accompanies this bump, so this is a plain buildViewerReadModel rebuild, not a data migration in the migrate-to-0.13.mjs sense — recorded here the same way migrate-to-0.10.mjs / migrate-to-0.13.mjs record their own one-off schema-bump fixes.
The directory tree navigates to /pages?directory=... but that filter has no control of its own on the Pages view (unlike urlPattern/status/etc., which show in the filter bar) — nothing on screen said which directory the list was scoped to. Add a one-line notice above the table when the filter is present, independent of column sort state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
directoryfilter is active, since it has no control of its own there (unlike the other filters shown in the filter bar)Performance fix (found via manual testing against a real ~450k-page archive)
/pages?directory=...exposed a pre-existing performance cliff: the legacydirectoryfilter is a substringLIKE '%dir%'againsturl_refs.url, which can't seek an index. Against the real archive, a directory-filtered request did not return within 2 minutes.viewer_pages.path_sort_keywas already stored for exactly this range-scan use case but had no index and no reader. Added thevp_pathindex and adirectoryoption tolistViewerPages/applyViewerPagesFilters— a>=/<range bounded by a maximal-codepoint sentinel, not a LIKE, so it matches the whole subtree while still seeking the index.VIEWER_READ_MODEL_SCHEMA_VERSIONto 18; existing archives fall back to the legacy path until their read model rebuilds (nitpicker viewer-build, or the includedscripts/migrate-to-0.14.mjs).Test plan
yarn lint— 0 errorsyarn build— all 13 packagesyarn test— full unit suite, 486 files / 3432 tests passedyarn workspace @nitpicker/viewer test:e2e:directory-tree— 12/12 passedyarn workspace @nitpicker/viewer test:e2e— 49/49 passed (regression check on the shared fixture)/api/pagesrequest went from >2min (no response) to ~84ms