Skip to content

Update packages installed for simulations - #143

Open
marySalvi wants to merge 1 commit into
mainfrom
simulation-patch
Open

Update packages installed for simulations#143
marySalvi wants to merge 1 commit into
mainfrom
simulation-patch

Conversation

@marySalvi

Copy link
Copy Markdown
Collaborator

I found that when running PHOLD I needed to add the libmpich-dev. Please check by spinning up a container without and running PHOLD. Then spin up a new container with this branch.

@marySalvi
marySalvi requested review from bnmajor and waxlamp July 21, 2026 18:13
@marySalvi marySalvi changed the title Add libmpich-dev for simulations Update packages installed for simulations Aug 5, 2026
@marySalvi
marySalvi requested review from bnmajor and waxlamp and removed request for bnmajor and waxlamp August 5, 2026 19:03
@bnmajor

bnmajor commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

@marySalvi were you able to test this and successfully complete a run with results? I am running into issues that appear to be related to mpich in docker, but I want to make sure you're having issues too before I go down that rabbit hole...

@marySalvi

marySalvi commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

were you able to test this and successfully complete a run with results

@bnmajor Yes, I am able to get it to run. What kind of errors are you getting?

@bnmajor

bnmajor commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Noting here that @marySalvi and I were able to reproduce the bug that I was seeing.

Bug

With mpich the run reports COMPLETED but produces zero or partial simulation output. Nothing errors - mpirun -np 4 phold ... exits cleanly and looks successful, but each rank silently falls back to running sequentially, instead of joining one coordinated multi-rank job. So instead of one real 4-rank simulation, you get 4 independent, non-communicating sequential simulations, each writing its own output to a separate directory. Our output-file picker only looks at one of those directories, so the run appears "completed" while the actual output is missing/incomplete.

Root cause

The mpich package is set up in a way that doesn't play well with how processes coordinate with each other inside Docker containers. In practice, the different MPI ranks never actually connect to form one job and MPICH doesn't warn us when this happens, so the failure is invisible until you notice the output is missing or incomplete.

I confirmed that OpenMPI does not have this problem - ranks coordinate correctly, with real inter-rank network events and complete PE/KP/LP record ingestion.

Suggestions

We should use OpenMPI as the MPI implementation used to run PHOLD.

Instead of moving forward with this PR, my suggestion is to go with the approach in #149. That branch builds PHOLD from source in a dedicated Docker build stage, compiling it against OpenMPI and baking the resulting binary directly into the image rather than relying on a host-mounted ROSS volume. Because the binary is built and run against the exact same single MPI toolchain, there's no possibility of a mismatch between the what PHOLD was linked against and whatever happens to be installed in the container (which is the root of the issue we're seeing here). It also adds a safeguard that fails a run outright when no output files are produced, instead of silently marking it COMPLETED, so even if a similar coordination failure were to happen again it the failure would surface immediately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants