Slow testing and CICD

[dylanjmcconnell]

Testing is rather slow

• Full uv run --frozen pytest took over 19 minutes on my machine (on python 3.12)
• CICD on github takes about the same amount of time

Where the the time is going

Seems most of the time is the the test_packaged_table_configs_for_each_version (make sense):

• 5 workbooks × ~290 odd tables took roughly ~16 min.

This will presumably grow overtime as more workbooks added.

Drilling into that a bit more

• each table triggers 3x pd.read_excel (table read, plus the left/right column checks).
• Some sheets are re-parsed a lot (e.g. the flow path forecast table has ~120 odd tables).
• --cov-branch adds a chunk time too

Potential options

• Switch to calamine engine (rust based alternative to openpyxl) - could more than halve time?
• Could potentially parallize some of the tests (pytest-xdist) since workbook configs are are independent.
• Large refactor (probably not worth it) to load and parse sheets only once. Lot of the parsing is then redundant. Also maybe not necessary if switch to calamine.
• dropping --cov-brnach or only running that on one of the jobs (doesn't make sense to run that on all 15 os/py versions).