feat(kcl-corpus): extract KCL into a training corpus - #40
Conversation
Adds @foundry/kcl-corpus: pure extractors that turn FOUNDRY's KCL sources into content-addressed records, behind a CorpusSinkPort so the Supabase adapter is the only file importing a vendor SDK. Covers repo constants, external sample sets, and project CAD workspaces. DesignDoc extraction reads components[] and never CadDoc.script, which is a compat mirror of the active component and would double-count it. Stock starter content is hashed once and flagged so it can be held out. Project/branch ids are HMAC-pseudonymized before reaching the sink.
Documents the two extraction traps (the CadDoc.script compat mirror, and DesignDoc holding head state only) and states the governance bar that gates chat/prompt extraction.
|
Closing — superseded by Per Alan's call, the corpus work shouldn't live in FOUNDRY: the codebase gets too big, and this needs to run on its own schedule against its own warehouse rather than inside the app. It's now a standalone CLI in atlas-ingest, with the Zoo generator and Supabase writer each behind a port, its own Nothing is lost by closing this — the extractors and the two KCL traps this PR documented ( |
1 participant
Adds
@foundry/kcl-corpus: extraction of FOUNDRY's KCL into a content-addressedcorpus for model training. Opening this for review rather than pushing to main
— it collects product data, so it should be a deliberate call, not a surprise.
Why a package rather than a script
AGENTS.mdrule 2 keeps vendor SDKs out of domain logic. Extractors here arepure functions over rows; everything writes through
CorpusSinkPort, and theSupabase adapter in
./sinkis the only file importing a vendor SDK. Swappingin a file dump or a dry run is a sink swap.
Extraction reuses
normalizeCadDoc,parseKclModuleImports,parseForeignImports, andparseCadParamsfrom@foundry/cadrather thanre-implementing them, so the corpus cannot drift from what the app believes a
CadDoc means.
Two traps this encodes
CadDoc.scriptis a compat mirror of the active component. Counting itwould duplicate one component per document, so
extractDesignDocreadscomponents[]only. There is a test pinning this.DesignDocis last-write-wins (@@unique([projectId, branchId, kind])),so it holds head state with no revision history. Every intermediate revision
is lost unless it passed through chat. Worth considering an append-only
CadRevisiontable separately — it is the highest-leverage change fortraining-data volume.
What it does NOT collect
Prompts and other user-authored text are not extracted. That step is gated
on a legal basis for training use (terms coverage and/or a workspace opt-out)
plus redaction of emails, URLs, tokens, and names. The extractor for it is
deliberately absent, not merely disabled.
Project and branch ids are HMAC-pseudonymized before reaching the sink, with the
salt held outside the corpus database.
Storage
Target is a separate Supabase project, not FOUNDRY's own database. The corpus
lives in its own
kclschema, not exposed through PostgREST, RLS enabled withno permissive policies — service-role only. Migration is checked in at
packages/kcl-corpus/migrations/0001_kcl_schema.sql.Phase 1 (repo constants) has been run end-to-end against that project: 4
snippets, 4 occurrences, re-scrape verified idempotent, all three export views
queryable.
Validation
pnpm lint,pnpm format:check,pnpm typecheck(16/16),pnpm test(11/11packages) all pass. 18 new tests.
Not done here
extractDesignDocis implemented and tested against synthetic docs, including alegacy v4 migration, but has not been run against a live database — none was
available.