Skip to content

fix(seed): load reference data even when migrations seeded a term - #869

Merged
jirhiker merged 1 commit into
stagingfrom
fix/seed-reference-data-completeness
Aug 21, 2026
Merged

fix(seed): load reference data even when migrations seeded a term#869
jirhiker merged 1 commit into
stagingfrom
fix/seed-reference-data-completeness

Conversation

@jirhiker

Copy link
Copy Markdown
Member

What broke

The development-mode startup seed (core/app.py lifespan → seed_all) calls ensure_seed_prereqs, which loaded core/lexicon.json only when lexicon_term was completely empty:

has_lexicon = s.scalar(select(LexiconTerm.id).limit(1)) is not None
if not has_lexicon:
    init_lexicon()

Since b2c3d4e5f6a7_transducer_data_maturity the migration inserts a lexicon term of its own. On a fresh database, migrations alone now leave exactly one row in lexicon_term — measured, not guessed — so that emptiness check reports "already loaded" and the real 1144-term lexicon is skipped entirely.

The fallout, in order:

  1. init_parameter violates parameter_default_unit_fkey, because ft and dimensionless were never inserted as lexicon terms. Two parameters fail and the errors are printed, not raised.
  2. seed_all calls random.choice(organization_terms) on an empty organization category → IndexError: Cannot choose from an empty sequence.
  3. Application startup failed. Exiting.

The frontend felt this as a CI outage: OcotilloUI's Cypress job polls http://localhost:8000/docs for the API, which was never going to answer, so the job burned its full 720s and failed with exit 124. See DataIntegrationGroup/OcotilloUI#350.

The fix

  • ensure_seed_prereqs now runs init_lexicon() and init_parameter() unconditionally. Both already leave existing rows alone — init_lexicon upserts categories, terms, and associations with on_conflict_do_nothing — so the emptiness shortcut was buying nothing and cost correctness.
  • init_parameter skips (parameter_name, matrix) pairs that already exist, so a re-run no longer trips uq_parameter_name_matrix and prints a wall of caught DatabaseErrors.
  • New assert_lexicon_ready fails with the empty categories named, rather than leaving the next contributor to decode an IndexError thrown deep inside the seed.

Verification

Against a fresh PostGIS database, recreating the CI sequence (drop schema → alembic upgrade headseed_all):

  • With this branch: lexicon_term rows after migrations only = 1, then the seed loads all 1144 and completes — contacts=2, SEED OK.
  • With staging's versions of the two files restored: the exact CI failure reproduces, same two FK violations on ft and dimensionless, same IndexError: Cannot choose from an empty sequence.

Tests: 4 new cases in tests/test_seed_prereqs.py covering that both initializers run even when the tables hold rows, that assert_lexicon_ready names the empty categories, that every required category has terms once reference data loads, and that init_parameter is idempotent.

uv run pytest --ignore=tests/transfers → 664 passed, 46 skipped, 6 xpassed. One unrelated failure, tests/test_ogc.py::test_ogc_project_areas_items_expose_groups_with_project_areas, reproduces identically on unmodified staging on this machine and is green on CI runners — local environment (arm64 host, amd64 PostGIS image), not this change. black and the pre-commit flake8 selection are clean.

🤖 Generated with Claude Code

The development-mode startup seed calls ensure_seed_prereqs, which loaded
core/lexicon.json only when lexicon_term was completely empty. Since
b2c3d4e5f6a7 (transducer data maturity) the migration inserts a lexicon
term of its own, so on a fresh database that emptiness check now sees one
row and skips the real lexicon entirely.

What followed: init_parameter tripped parameter_default_unit_fkey because
'ft' and 'dimensionless' were never inserted, then seed_all called
random.choice on an empty organization category and died with
"IndexError: Cannot choose from an empty sequence". FastAPI aborted
startup, so the frontend's Cypress job sat on a readiness probe for its
full 720s timeout and failed with exit 124.

Both initializers leave existing rows alone, so ensure_seed_prereqs now
just runs them. init_parameter skips names already stored rather than
letting every re-run trip the unique constraint, and a new
assert_lexicon_ready names the empty categories instead of failing deep
inside the seed with an opaque IndexError.

Verified against a fresh database: migrations alone leave exactly one
lexicon term, and the seed now loads all 1144 and completes.
@github-actions

Copy link
Copy Markdown
Contributor

Coverage

79.05% total — gate is 75%.

Coverage for the Python files changed in this PR
Name Stmts Miss Cover Missing
core/initializers.py 137 20 85% 67-69, 73-82, 184, 188, 206, 271-284, 288-289
TOTAL 137 20 85%

@jirhiker
jirhiker merged commit cdb4246 into staging Aug 21, 2026
9 checks passed
@jirhiker
jirhiker deleted the fix/seed-reference-data-completeness branch August 21, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant