Patch delays - #148
Open
YaphetKG wants to merge 9 commits into
Open
Conversation
Pairs with the dug change that makes expand_concept fetch concurrently. That bounds out at roughly the number of real TranQL queries per concept, measured at ~4, so the rest of the concurrency has to come from here: the crawl_tranql loop over concept files, which is one iteration per annotated input and tens of thousands of them for a dbGaP dataset. Files are independent -- each decodes its own concepts, expands them and writes its own output dir -- and the work is nearly all http wait, so threads scale it despite the GIL. The loop body moves to crawl_one_file unchanged. Failures are re-raised rather than left in the pool, because a file that raises produced no output and the task must not report success. indexing.crawl_workers and indexing.crawl_file_workers multiply, and their product should not exceed what TranQL can serve at once -- past that, requests only queue at its gunicorn. search-chart carried workerCount: 1 with a sync worker, which is why the service sat at 2.5% of its two-core limit while being the bottleneck; that goes to 16 alongside this. The crawlspace wiring goes away with the file cache it fed: crawl_concepts no longer creates or assigns it, and crawl_tranql no longer clears the crawl_output dir, since nothing writes there now.
A bare carriage return with no paired \n still reads as a line break under universal newlines, so both our own tooling and the falkordb_bulk_loader CSV parser split one row into two, corrupting the column count. Hit in production on a dbGaP Study "activitybk" field whose description had codebook text pasted in with stray CRs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
expanded_concepts.txt and the per-directory expanded elements.txt embed full TranQL knowledge-graph subgraphs per concept -- repeated CURIEs, biolink categories, and JSON keys that compress 5-10x+. That's the difference between a large dataset's crawl output fitting on the task PVC and hitting ENOSPC mid-run, as bdc-recover just did at 25h/try 12. Same filename/extension as before, so every existing glob pattern that finds these files by name still matches. read_object detects the gzip magic number, so artifacts already committed to lakefs before this still read as plain text -- no forced re-migration. migrate_pickled_classes.py reads/writes these same files directly (bypassing storage.py), so it needs the same transparent gzip handling to keep working -- and now also brings old plain-text artifacts up to the current (gzip) on-disk format when it rewrites them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gzip.compress's default (level 9) cost 3-4x the CPU of level 6 for the same compressed size on this repetitive JSON -- measured identical output bytes at both levels on realistic KG-answer-shaped data. Deadly under a crawl task's thin CPU limit: bdc-recover's pod (250m limit, 4 crawl workers) was throttled 70% of scheduling periods after the gzip change landed, cutting throughput from ~4 dirs/min to ~3 dirs/min. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
crawl_file_workers threads do real CPU work now (TranQL response handling, jsonpickle encode, gzip) -- measured a crawl pod throttled 70% of its scheduling periods on the chart's default cpu limit with 4 worker threads, cutting throughput to roughly a third of what it should be. New indexing.crawl_cpu config (default "1", a full core) plumbed through a generalized resource_override (memory_override is now a thin wrapper over it) the same way annotate_memory already overrides the crawl task's memory limit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 core still left 4 crawl_file_workers threads sharing it. Match cores 1:1 with worker threads instead. Plenty of quota headroom in both live namespaces to absorb it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
crawl_one_file had no resume check, unlike annotation_is_complete for annotate -- a killed/retried crawl re-walked every file from the start of the sorted list. TranQL's response cache makes redoing an already-crawled file cheap, but not free, and a large dataset's never-before-seen files queue behind all of that free-but-not-instant rework before any real new progress resumes. Mirrors the annotate pattern: crawl_output_path/crawl_is_complete check expanded_concepts.txt existence, crawl_tranql filters pending files down before submitting them to the pool. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… output crawl_bdc-parent died with ENOSPC downloading its own input: bdc-parent's elements.txt/concepts.txt were annotated before storage.write_object started gzipping .txt/.jsonl artifacts, so they're still ~3MB/dir raw -- 61,597 dirs is ~186GB before crawl writes a single byte of its own (gzipped) output. Pure byte-level gzip, no jsonpickle decode/re-encode like migrate_pickled_classes.py does -- content is untouched, just smaller, and there's no class-path migration needed here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.