The loader's merge_semantic_meta_data LPUSHes into the semantic_types key, and because that function runs once per loader Kubernetes Job (mode: load shards one Job per file), the list ends up with one copy of every type per file — dozens of duplicates.
The frontend papers over this by deduping on read (get_semantic_types uses list(set(...)), and get_curie_prefixes now does the same as of the loader-reorg branch). The proper fix is to switch the key to a Redis SET (SADD in the loader, SMEMBERS in the frontend).
Why this is not a drop-in change: mode: restore loads pre-built RDB backups produced by the old loader, where semantic_types is a LIST. A new frontend doing SMEMBERS against a restored LIST gets WRONGTYPE, breaking /get_semantic_types and /get_curie_prefixes. This needs a coordinated loader + frontend + backup-regeneration rollout.
Context: documentation/Development.md (backlog section), node_normalizer/loader/loader.py merge_semantic_meta_data, node_normalizer/normalizer.py get_curie_prefixes.
The loader's
merge_semantic_meta_dataLPUSHes into thesemantic_typeskey, and because that function runs once per loader Kubernetes Job (mode: loadshards one Job per file), the list ends up with one copy of every type per file — dozens of duplicates.The frontend papers over this by deduping on read (
get_semantic_typesuseslist(set(...)), andget_curie_prefixesnow does the same as of the loader-reorg branch). The proper fix is to switch the key to a Redis SET (SADDin the loader,SMEMBERSin the frontend).Why this is not a drop-in change:
mode: restoreloads pre-built RDB backups produced by the old loader, wheresemantic_typesis a LIST. A new frontend doingSMEMBERSagainst a restored LIST getsWRONGTYPE, breaking/get_semantic_typesand/get_curie_prefixes. This needs a coordinated loader + frontend + backup-regeneration rollout.Context:
documentation/Development.md(backlog section),node_normalizer/loader/loader.pymerge_semantic_meta_data,node_normalizer/normalizer.pyget_curie_prefixes.