Add TorsionNet500CCSDT benchmark visualisation, MAE metric, and docs - #682
Add TorsionNet500CCSDT benchmark visualisation, MAE metric, and docs#682ellalbolland wants to merge 6 commits into
Conversation
|
Thanks @ellalbolland, from a brief scan its looking great! For the uma model access you can request it i think here: https://huggingface.co/facebook/UMA
|
ElliottKasoar
added
the
new benchmark
|
Here's the input dataset used for the calc: TorsionNet500_xyz (1).zip |
thanks uploaded |
|
Hi @ellalbolland, everything is looking really nice so far! One thing that would be great to add is visualisaiton from the torsion curve plot. you could take a look at some of the NEB curves to see how they use the trajectory mode to visualise data points along the curve. thanks! |
|
the visualisaition is looking great! @ElliottKasoar could you take a look when you have time |
Adds per-fragment RMSE/MAE scatter plots and per-fragment torsion energy profile curves to the TorsionNet500CCSDT analysis, wired into the app via a table-cell -> scatter -> curve click-through. Renames the raw energy fields to ref_energy/model_energy (and ref_rel_energy/model_rel_energy for the mean-centered profiles) to match existing codebase conventions, and switches to the shared rmse/mae utilities instead of a local reimplementation. Named TorsionNet500CCSDT rather than TorsionNet500 to coexist alongside the existing barrier-height implementation in ddmms#642/ddmms#659, since this benchmark reports RMSE/MAE of the full relative energy profile against CCSD(T)/CBS reference data - a different metric and reference level of theory, not a duplicate.
Save per-fragment elemental info via get_struct_info so the app can filter TorsionNet500CCSDT by element, and set precision="high" explicitly on the calculator.
…CSDT.py Co-authored-by: Joseph Hart <92541539+joehart2001@users.noreply.github.com>
…CSDT.py Co-authored-by: Joseph Hart <92541539+joehart2001@users.noreply.github.com>
Clicking a point on a torsion curve now shows the corresponding 3D structure at that dihedral angle, using WEAS trajectory mode, matching the pattern already used for NEB benchmarks.
6d92910 to
66474c0
Compare
ElliottKasoar
left a comment
ElliottKasoar
left a comment
There was a problem hiding this comment.
This looks really nice, thanks @ellalbolland! I believe @lwehrhan proposed closing #659 and adding their data to this, so perhaps we should just rename the benchmark TorsionNet500?
Otherwise just a couple of minor suggestions/questions
I just rebased and fixed the conflict in the docs, hopefully that's ok
| xyz_files = sorted(model_dir.glob("*.xyz")) | ||
|
|
||
| for xyz_file in xyz_files: | ||
| atoms = read(xyz_file, ":") |
There was a problem hiding this comment.
We may want to be a bit careful here, as there's nothing to check if some of the data is missing, as far as I can tell. For example, I interrupted one of my models after it only ran a couple of structures, and it got a score, presumably based on the average of those.
We do want (1) this to not break if data is missing and (2) to be flexible enough that when we do "full" filtering, we can return the full list of errors that were calculated, but we want to make sure a model has all of the errors that we expect, and give a None/NaN score in if not.
One way to do that is to use the labels/filename stems from get_struct_info, but any way to do this would be good.
In practice, I didn't see any models fail calculations, so the only real risk is if a calculation gets interrupted, but we should catch this.
| """ | ||
| Attach callbacks that show a torsion curve and structure on point clicks. | ||
|
|
||
| Unlike `plot_from_scatter`, this reads the clicked fragment's curve JSON from |
There was a problem hiding this comment.
Is the only difference the lazy loading? It might be worth just doing this for all plots from scatters, or at least combining these with a lazy_load flag
3 participants
Pre-review checklist for PR author
Summary
Adds per-fragment RMSE/MAE scatter plots and per-fragment torsion energy profile curves to the TorsionNet500CCSDT analysis, enabling table-cell → scatter → curve click-through navigation.
Named TorsionNet500CCSDT rather than TorsionNet500 to coexist alongside the existing barrier-height implementation in #642/#659, since this benchmark reports RMSE/MAE of the full relative energy profile against CCSD(T)/CBS reference data - a different metric and reference level of theory, not a duplicate.
Linked issue
Resolves #679
Progress
Testing
Tested against 16 of the 21 registered models (full 500/500-fragment recompute completed for each). The uma family (uma-s-1p1-omat, uma-m-1p1-omat, uma-s-1p1-omol, uma-s-1p2-omol, uma-m-1p1-omol) could not be run due to lacking HuggingFace authorisation for the gated UMA model weights.
Confirmed working end-to-end by running the app standalone (
uv run python ml_peg/app/conformers/TorsionNet500CCSDT/app_TorsionNet500CCSDT.py) and clicking through the table -> scatter -> curve interactivity manually.New decorators/callbacks
One new local callback,
_register_curve_callbackinapp_TorsionNet500CCSDT.py.plot_from_scatterpre-loads every possible figure upfront (too expensive for 500 curves/model);struct_from_scatterlazy-loads but only for structures, not Plotly figures. This one lazy-loads a curve JSON from file per click instead. Flagged with aTODOto move it intobuild_callbacks.pyif another benchmark needs the same pattern.