Performance improvements planning

[John-F-Wagstaff]

Task 1 - Liftover

• in the case of genomic liftover via transcript mappings we make no attempt to preference the ungapped alignments. Fixing this would reduce the overhead of liftovers where both gapped and ungapped mappings are available. ( This will increase the complexity of the liftover transcript selection code, and won't have an affect on the simpler use case, but should dramatically speed up the process, and possibly improve accuracy, when it does trigger.)

• A preference for MANE Select could be added at the same time and might be useful for reliability (or at least consistancy)

• If this is implemented we could also use this as a foundation to improve liftover granularity (e.g. only use transcript liftover when non-gapped transcript mappings are available, or don't liftover at all when only gapped lifts are possible).

[Peter-J-Freeman]

Task 2- Regex helper functions

Background

Currently, the code uses multiple Regexes to spot data that can be derived directly from hgvs python objects.

This is very slow as we need to

• String an hgvs object (or object attribute) - Takes time
• Compile a regex pattern - Takes time
• Do the regex search - Takes time
• Sometimes, re-parse the string into an object again post-manipulation - Takes a lot of time

Solution

Develop and plug in a range of helper functions into a new file or into utils.py. I think I prefer a new file e.g. object_parse_utils.py

Intronic positions

• Start position is intronic
• Start position is exonic
• End position is intronic
• Start position is Exonic
• A function to tie these all together to assess both start and end

UTR positions

• Start position is UTR
  • and which UTR
• End Position is UTR
  • and which UTR
    Both positions are UTR
  • and which UTRs

@John-F-Wagstaff, any helpers I missed?

Benchmarking

Before updating the code, run Pytest 3 or 4 times and log the speed in seconds
Do the same after updating (and in stages throughout) to log progress.

We suspect that we could see a distinct increase in speed

[Peter-J-Freeman]

Task 3- Update Variant Formatter

• Variant Formatter needs to be updated to use the VV object creation rather than reparsing
• The helper functions in Task 2 need to be integrated
• @John-F-Wagstaff, anything I missed?

[John-F-Wagstaff]

Task 4- Explore detecting alignment gaps near variants

• check complexity of extracting the near gap/not near gap status for variants in gap alignments, if possible this allows us to skip the push-left/right functions for the majority of gapped variants
• check the complexity of using the near gap status of the variant to loop through nearby indels when pushing left&right, as opposed to looping one base at a time (usually this would switch us to only 1 check for gap neighbouring variants that might normalise in, as opposed to 1 per base pair)

Unfortunately these touch some of our most fragile and complex code but could both easily reduce the time taken for currently tested as gap relevant variants by an order of magnitude, perhaps more.

[Peter-J-Freeman]

Task 5 - Error code mapping

Go through the code base and ensure every warning and Error has an error code.

• CamelCaseCode: Message
• Message must have at least 3 words before a variable, in case we still need to use the mapper.
• vv_hgvs messages will also need to be updated to this format and VV updated to handle the updates.
• VF will also need updating and Error Codes added

[ifokkema]

Regarding task 1: is this related to the problem that I have, where I sometimes get multiple liftovers back when I have a genomic variant mapped to multiple transcripts? If so, that would be great, because I'm currently considering dropping these variants from my pipeline as I cannot reliably lift them over... although I guess I could simply use the value returned when using the MANE Select transcript?

[John-F-Wagstaff]

Yes and no, unfortunately, it is related, but the first stage of task 1 would not have an affect, and the final one would only have an affect, if you tried genomic liftover with the right settings when you hit this kind of problem and did a comparison afterwords.

The most likely causes of this sort of issue is that the variant hits a gap (or in principal just a mismatch) in the transcript to/from genome mapping for at least one of the genomic targets, but differs for different transcripts. This could be a gap in the mapping for one or several transcript genome pairs, or it could be a differing mapping location for different transcripts to the same genomic reference.

In the case when:

1. the mapping is ungapped for the both genomic mappings of at least one transcript (which could then be detected via a post inital mapping genomic liftover with the correct settings)
2. there are not multiple possible mapping locations via different transcripts that are each ungapped (current genomic liftover code would not detect this) or at least there is a MANE transcript and it can be considered reliable
3. the mapping differences are not a signe of a fundamentally unreliable not accurately mappable location/variant

We might be able to get good output, from a strict no-gap transcript based liftover. How to detect the third issue in the above list however is a whole other problem, this kind of thing is a bit of a warning sign for reliability.