Storing and using large artifacts

[dkoslicki]

It's been discussed a fair bit internally in the ARAX team, and a bit during Hackathons and the like with the Shepherd folk, but we have not tracked the issue of how to use large artifacts in the Shepherd framework. For example,

1. Our MVP1 approach uses a large database to quickly return pre-computed results
2. Our NGD values are semi-precomputed via stores of (term, PMIDs) tuples
3. Pathfinder would be enormously faster if it had access to a slimmed down, TSV-like representation of tier 0/1 graphs available in memory at runtime
4. etc. etc.

Perhaps this is worth discussing soon, and/or at the upcoming June hackathon.

Tagging @maximusunc , @tokebe , @chunyuma, and @mohsenht as that is a (subset) of stakeholders in this question

[maximusunc]

Sounds good, @dkoslicki! Maybe we can start information gathering. Can you share a list of each artifact you are currently using with some info on each?

Things I think would be useful to know about each artifact:

• What kind of artifact is it? sqlite, redis, text file?
• How big is it?
• How often is it hit/used per query? (i/o)
• Where is it being used? Do multiple parts of the workflow use it?

I think once we know all this, we can start thinking about hosting and scalability and how they can fit into Shepherd.

[dkoslicki]

@mohsenht Can you provide the artifact info for Pathfinder (noting the external SQLite server currently being hit)

@chunyuma Can you provide this info for xDTD (including things needed like the curie_to_PMID stuff)

@venkataseshtej if you are able to, can you provide this info for the current status (before your planned refactor) of xCRG? I know the transcription factor file(s) and saved model (RF or xGBoost?) are required, I just don't know their size.

But in summary @maximusunc , it's likely a gamut:

• What kind of artifact is it? sqlite, redis, text file?
  • sqlite, saved model files, text files
• How big is it?
  • from 10's of megs to multiple gigs
• How often is it hit/used per query? (i/o)
  • some hit at run time for every query (eg. MVP1/xDTD question is a lookup from a big pre-computed table; Pathfinder hits its SQLite DB multiple times during each expansion of its multi-hop expansion).
• Where is it being used? Do multiple parts of the workflow use it?
  • some hit at run time for every query, some only at re-build/re-train time

But I'll let the folks above chime in about the exact details

[mohsenht]

Following up on David’s note, Pathfinder currently relies on two specific remote MySQL databases:

Artifact Name: curie_ngd_v1_0_kg2_10_2
Artifact Type: MySQL (Remote)
600MB

Artifact Names: kg2c_v1_0_kg2_10_2
Artifact Type: MySQL (Remote)

3.5 GB for node degrees "tier 0 for overlay". This can be reduced as there's a bunch of other stuff in it

[chunyuma]

@chunyuma Can you provide this info for xDTD (including things needed like the curie_to_PMID stuff)

The new version of xDTD will not reply on any local large database, but it will have a large xDTD to store the pre-computed results.

To follow up on David’s note:

1. A sqlite format of xDTD database is required.
2. Its size might be ~40GB.
3. Each time a infer query is used, xDTD database will be hit once per query.
4. It is used with the infer mode.

Please let me know if you need more details about xDTD. Thanks!

[venkataseshtej]

@venkataseshtej if you are able to, can you provide this info for the current status (before your planned refactor) of xCRG? I know the transcription factor file(s) and saved model (RF or xGBoost?) are required, I just don't know their size.

xCRG (old) artifacts:

1. Embeddings: NumPy archive (chemical_gene_embeddings...npz), about 1.6 GB; loaded at initialization and used in every xCRG infer query; used in creativeCRG.py.
2. Increase model: Random Forest (xcrg_increase_model...pt_), about 1.3 GB; loaded at initialization and used in every xCRG infer query; used in creativeCRG.py.
3. Decrease model: Random Forest (xcrg_decrease_model...pt_), about 2.0 GB; loaded at initialization and used in every xCRG infer query; used in creativeCRG.py.
4. TF list: JSON (transcription_factors.json), about 13 KB; loaded at initialization and used for transcription-factor filtering or neighbor logic in creativeCRG.py.

Notes:

1. The .pt files are Random Forest models loaded via joblib, not PyTorch models.
2. Artifact paths are configured through config_dbs.json and RTXConfiguration.py.
3. All four artifacts are loaded into memory when chemical_gene_regulation_graph_expansion initializes xCRG; together they are ~5 GB on disk and require several GB of memory per process.