fix: paginate GitHub trees on truncation to avoid silently dropping files

[GoodbyePlanet]

Problem

list_github_files (server/indexer/github_source.py) fetched the entire repo tree in a single GitHub Trees API call with recursive=1. The Trees API has an undocumented response-size limit; on very large repos GitHub sets truncated: true and returns only a partial tree. semcode detected this but only logged a warning and then iterated over whatever partial tree it got back — so an unknown subset of files was silently absent from the index run (never parsed, embedded, or upserted, and thus unsearchable).

Closes #55

Fix

Keep the fast path (one recursive=1 call) for the common case. When the response is truncated, fall back to a recursive per-subtree walk using non-recursive git/trees/<sha> calls — each single-directory listing is tiny and effectively never hits the size limit. This reuses git blob SHAs directly (exactly what GitHubFile.blob_sha needs for incremental indexing), so no API-shape changes are required.

• _filter_tree_blobs() — extracted shared filter helper (root prefix / supported extension / exclude) so the fast path and fallback filter identically.
• _walk_tree_recursive() — walks git/trees/<sha> per directory, joins full paths from per-level entry names, prunes subtrees outside root, and fetches siblings concurrently via an asyncio.Semaphore (_TREE_WALK_CONCURRENCY = 10, mirroring _DIFF_CONCURRENCY). Seeds from the root tree SHA already present in the truncated response — no extra ref-resolution round-trip.
• _subtree_is_relevant() — the root-prefix pruning predicate.

Tests

New tests/test_github_source.py (respx-mocked):

• non-truncated filtering (supported ext, root, exclude)
• truncated fallback recovering a file the recursive response omitted, with correct full-path joining
• root-prefix pruning: subtrees outside root are never fetched

Full suite green (208 passed).

Docs

docs/ingestion.md — replaced the "Large-repo warning" (silent drop) note with a description of the new fallback.

🤖 Generated with Claude Code