The loader (node_normalizer/loader/loader.py) writes with individual SETs queued on a pipeline. For each compendium line that is N SETs into eq_id_to_id_db (one per equivalent identifier) plus one each into id_to_eqids_db / id_to_type_db / info_content_db; each conflation line is M SETs. Every compendium Job hammers the same single-threaded eq_id_to_id_db and id_to_eqids_db instances at once, so the command-processing cost on those two servers is the load's throughput ceiling (see documentation/Loading.md).
Batching these into MSET (accumulate a {key: value} map per block, flush with MSET, chunked to a sane size) cuts the number of commands those single-threaded servers must parse — the likely next win after disabling periodic saves and transaction=False (done in the load-faster PR).
Why this was held out of the 2.5 load-faster PR: it is the one change that could silently drop data if the accumulate/flush logic is wrong (e.g. a within-block key collision or a missed final flush), and the project owner is specifically risk-averse about silent inconsistency. It should ship with a correctness test that loads a fixture and asserts the loaded keys/values match expected, so a regression fails CI rather than surfacing as a quietly incomplete production database.
Context: load_compendium / load_conflation in node_normalizer/loader/loader.py; documentation/Loading.md.
The loader (
node_normalizer/loader/loader.py) writes with individualSETs queued on a pipeline. For each compendium line that is NSETs intoeq_id_to_id_db(one per equivalent identifier) plus one each intoid_to_eqids_db/id_to_type_db/info_content_db; each conflation line is MSETs. Every compendium Job hammers the same single-threadedeq_id_to_id_dbandid_to_eqids_dbinstances at once, so the command-processing cost on those two servers is the load's throughput ceiling (seedocumentation/Loading.md).Batching these into
MSET(accumulate a{key: value}map per block, flush withMSET, chunked to a sane size) cuts the number of commands those single-threaded servers must parse — the likely next win after disabling periodic saves andtransaction=False(done in the load-faster PR).Why this was held out of the 2.5 load-faster PR: it is the one change that could silently drop data if the accumulate/flush logic is wrong (e.g. a within-block key collision or a missed final flush), and the project owner is specifically risk-averse about silent inconsistency. It should ship with a correctness test that loads a fixture and asserts the loaded keys/values match expected, so a regression fails CI rather than surfacing as a quietly incomplete production database.
Context:
load_compendium/load_conflationinnode_normalizer/loader/loader.py;documentation/Loading.md.