Skip to content

Make post-refinement scalable #11

Description

@biochem-fan

At SACLA, we have only one fat-memory node (1TB RAM) and a time limit of 24 hr for a job. Thus, the memory and time requirement of post-refinement limits the size of the dataset we can process.

I have a suggestion to make post-refinement more scalable on multiple nodes.

  1. refine images in parallel on multiple worker nodes using a common reference
  2. write the refined parameters to the (shared) disk
  3. the master node waits for all worker nodes
  4. the master node merges all images using refined partialities and creates a new reference
  5. distribute the new merged reference to worker nodes and repeat from 1.

This can be ideally implemented using MPI, but more easily by shell scripts using GNU parallel with minimum modifications to cppxfel.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions