Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

moller (part of the ISSP HTP-Tools package) generates batch job scripts for supercomputers and clusters. It reads a YAML job description and emits a single bash script that runs a chain of tasks over many dataset directories in parallel using GNU Parallel, with per-task status logging, resume/retry support, and dependency skipping between tasks.

## Commands

```bash
# Install (poetry-core build; editable install works too)
python3 -m pip install .

# Generate a job script from a YAML description (writes to stdout without -o)
moller input.yaml -o job.sh

# Show job status from GNU Parallel joblogs (stat_<task>.dat) in the run directory
moller_status input.yaml list.dat # optional: --csv/--html, --failed/--ok/--yet/...

# Build docs (Sphinx, both languages; requires sphinx installed)
make -C docs/en html
make -C docs/ja html
```

There is no pytest suite and no linter configured. `tests/moller/` is an on-cluster integration test: `mkdataset.sh` creates dataset directories and `list.dat`, then the moller-generated `job.sh` is submitted to run `moller_test.py` (an MPI program with configurable sleep/failure rate) — it can only be validated on a machine with a scheduler and mpi4py. Sample YAML inputs in `sample/moller/` and `tests/moller/input.yaml` are the reference for input-format behavior; `reference/` subdirectories hold expected status output.

Versioning: `src/moller/__init__.py` (`__version__`) and `pyproject.toml` must be kept in sync. Development flows through the `develop` branch and merges to `main`; docs are auto-built and deployed to gh-pages by `.github/workflows/deploy_docs.yml`.

## Architecture

Two entry points (declared in pyproject.toml):

- `moller.main:main` — script generation. `ScriptGenerator` parses the YAML and assembles an ordered task list: prologue → bash function definitions → logfile check → jobs (in YAML order) → epilogue. Each entry in `jobs:` becomes either a `TaskParallel` (default) or a `TaskSerial` (`parallel: false`). A `TaskParallel` is emitted as a bash function `task_<name>` invoked through GNU Parallel over the work items piped in from `list.dat`, logging to `stat_<name>.dat`.
- `moller.moller_status:main` — status reporting. Re-parses the same input YAML to discover the parallel tasks and their `stat_<task>.dat` joblogs, then renders a job × task table (o = ok, x = failed, `-` = skipped, `.` = not run) as text/csv/html with filters.

Platform abstraction (`src/moller/platform/`):

- `base.py` defines the `Platform` base class plus a factory registry: `register_platform(name, cls)` / `create_platform(name, info)`. Each concrete platform module calls `register_platform(...)` at import time, and `platform/__init__.py` imports them all — a new platform is not usable until it is imported there.
- Scheduler families are the mid-layer bases: `base_slurm.py` (Slurm/#SBATCH), `base_pbs.py` (PBS/#PBS), `base_default.py` (plain bash, no scheduler). Concrete platforms are thin subclasses: `ohtaka` (Slurm, ISSP), `kugui` (PBS, ISSP), `pbs`, `default` — typically overriding only `parallel_command()` and defaults.
- `function.py` holds the scheduler-independent bash helpers embedded into every generated script (`run_parallel`, `_is_ready`, retry/ulimit handling, DEBUG); scheduler-specific helpers (`_setup_taskenv`, `_setup_run_parallel`, `_find_multiplicity`) are defined in the `base_slurm`/`base_pbs`/`base_default` classes and assembled by each base's `generate_function()`.

Key generation mechanics to preserve when editing:

- In each parallel task's `run:` block, any line containing `srun`, `mpirun`, or `mpiexec` has that word substituted with the platform-specific parallel command (e.g. ohtaka: `srun --exclusive --mem-per-cpu=1840 -N $_nn -n $_np -c $_nc`). The `node:` spec (scalar or 1–3 element list) is normalized to `[nodes, procs, cores]` and passed as the `$_sig` signature to `run_parallel`, which computes how many work items run concurrently.
- Task chaining: consecutive parallel tasks are linked via `prev_log_file` — the generated `_is_ready` check reads the previous task's joblog and exits 255 for work items whose previous step failed, which `moller_status` reports as skipped (`-`). Serial tasks do not participate in this chain.
- Joblog format is GNU Parallel's tab-separated joblog; `moller_status.read_joblog_file` parses the `Command` field positionally (4 fields = single parallel, 7 = nested parallel) — changes to the `run_parallel` invocation signature in generated scripts must keep that parser in sync.
61 changes: 57 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
<div align="center">
<img src="docs/images/moller_logo.png" alt="moller logo" width="200">
</div>

# moller

[![Release](https://img.shields.io/github/v/release/issp-center-dev/Moller)](https://github.com/issp-center-dev/Moller/releases)
[![License: GPL-3.0-or-later](https://img.shields.io/badge/License-GPL--3.0--or--later-blue.svg)](https://www.gnu.org/licenses/gpl-3.0.en.html)
[![Docs](https://github.com/issp-center-dev/Moller/actions/workflows/deploy_docs.yml/badge.svg)](https://github.com/issp-center-dev/Moller/actions/workflows/deploy_docs.yml)
[![Python](https://img.shields.io/badge/Python-3.7%2B-blue.svg)](https://www.python.org/)
[![DOI](https://img.shields.io/badge/DOI-10.1080%2F27660400.2025.2564055-blue)](https://doi.org/10.1080/27660400.2025.2564055)

In recent years, the use of machine learning for predicting material properties and designing substances (known as materials informatics) has gained considerable attention.
The accuracy of machine learning depends heavily on the preparation of appropriate training data.
Therefore, the development of tools and environments for the rapid generation of training data is expected to contribute significantly to the advancement of research in materials informatics.
Expand All @@ -9,8 +19,23 @@ It is a tool for generating batch job scripts for supercomputers and clusters, a

## Supported platforms

- ISSP supercomputer systems: ohtaka, kugui
- general cluster machines and workstations
moller generates batch job scripts for the following types of platforms:

- ISSP supercomputer systems: ohtaka (slurm), kugui (PBS)
- generic PBS-based clusters
- general cluster machines and workstations without a job scheduler

### Pre-installed systems

moller is available on the following supercomputer systems.
For instructions on using moller on each system, please refer to its user guide.

| System | Operated by | Status |
| --- | --- | --- |
| ohtaka, kugui | [Supercomputer Center, ISSP](https://mdcl.issp.u-tokyo.ac.jp/scc/en/), The University of Tokyo | pre-installed |
| [AOBA](https://www.cc.tohoku.ac.jp/english/) | Cyberscience Center, Tohoku University | pre-installed |
| [Genkai](https://www.cc.kyushu-u.ac.jp/scp/en/system/genkai/) | Research Institute for Information Technology, Kyushu University | pre-installed |
| [Miyabi](https://www.cc.u-tokyo.ac.jp/en/supercomputer/miyabi/service/) | Information Technology Center, The University of Tokyo | supported; pre-installation in progress (as of Aug 14, 2026) |

## Requirement

Expand All @@ -27,15 +52,43 @@ python3 -m pip install DIRECTORY_OF_THE_REPOSITORY
## License

The distribution of the program package and the source codes for moller follow
GNU General Public License version 3
([GPL v3](https://www.gnu.org/licenses/gpl-3.0.en.html)).
GNU General Public License version 3 or later
([GPL-3.0-or-later](https://www.gnu.org/licenses/gpl-3.0.en.html)).

Copyright (c) <2023-> The University of Tokyo. All rights reserved.

This software was developed with the support of
"Project for Advancement of Software Usability in Materials Science"
of The Institute for Solid State Physics, The University of Tokyo.

The installation of moller on Miyabi and Genkai was supported by
JST Moonshot R&D Program (Grant Number JPMJMS24A3).

## Citation

When publishing results obtained using this software, we would appreciate it if you cite the following paper:

> Kazuyoshi Yoshimi, Yuichi Motoyama, Tatsumi Aoyama, Mitsuaki Kawamura, and Naoki Kawashima,
> "Project for advancement of software usability in materials science",
> Science and Technology of Advanced Materials: Methods **5**, 2564055 (2025).
> [https://doi.org/10.1080/27660400.2025.2564055](https://doi.org/10.1080/27660400.2025.2564055)

BibTeX entry:

``` bibtex
@article{Yoshimi2025,
author = {Kazuyoshi Yoshimi and Yuichi Motoyama and Tatsumi Aoyama and Mitsuaki Kawamura and Naoki Kawashima},
title = {Project for advancement of software usability in materials science},
journal = {Science and Technology of Advanced Materials: Methods},
volume = {5},
number = {1},
pages = {2564055},
year = {2025},
doi = {10.1080/27660400.2025.2564055},
url = {https://doi.org/10.1080/27660400.2025.2564055}
}
```

## Official page

- [HTP-tools project page](https://www.pasums.issp.u-tokyo.ac.jp/htp-tools/)
Expand Down
69 changes: 60 additions & 9 deletions docs/en/source/moller/about/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@ Therefore, the development of tools and environments for the rapid generation of

moller is provided as part of the HTP-Tools package, designed to support high-throughput computations.
It is a tool for generating batch job scripts for supercomputers and clusters, allowing parallel execution of programs under a series of computational conditions, such as parameter parallelism.
Currently, it supports the supercomputers ohtaka (using the slurm job scheduler) and kugui (using the PBS job scheduler) provided by the Institute for Solid State Physics, University of Tokyo.
Currently, it supports the supercomputers ohtaka (using the slurm job scheduler) and kugui (using the PBS job scheduler) provided by the Institute for Solid State Physics, University of Tokyo, as well as generic PBS-based clusters and workstations without a job scheduler.

License
----------------------------------------------------------------

The distribution of the program package and the source codes for moller follow GNU General Public License version 3 (GPL v3) or later.
The distribution of the program package and the source codes for moller follow GNU General Public License version 3 or later (GPL-3.0-or-later).

Contributors
----------------------------------------------------------------
Expand All @@ -31,25 +31,25 @@ This software was developed by the following contributors.

- Developers

- Kazuyoshi Yoshimi (The Instutite for Solid State Physics, The University of Tokyo)
- Kazuyoshi Yoshimi (The Institute for Solid State Physics, The University of Tokyo)

- Tatsumi Aoyama (The Instutite for Solid State Physics, The University of Tokyo)
- Tatsumi Aoyama (The Institute for Solid State Physics, The University of Tokyo)

- Yuichi Motoyama (The Instutite for Solid State Physics, The University of Tokyo)
- Yuichi Motoyama (The Institute for Solid State Physics, The University of Tokyo)

- Masahiro Fukuda (The Instutite for Solid State Physics, The University of Tokyo)
- Masahiro Fukuda (The Institute for Solid State Physics, The University of Tokyo)

- Kota Ido (The Instutite for Solid State Physics, The University of Tokyo)
- Kota Ido (The Institute for Solid State Physics, The University of Tokyo)

- Tetsuya Fukushima (The National Institute of Advanced Industrial Science and Technology (AIST))

- Shusuke Kasamatsu (Yamagata University)

- Takashi Koretsune (Tohoku University)

- Project Corrdinator
- Project Coordinator

- Taisuke Ozaki (The Instutite for Solid State Physics, The University of Tokyo)
- Taisuke Ozaki (The Institute for Solid State Physics, The University of Tokyo)


Copyright
Expand All @@ -67,10 +67,61 @@ Copyright

This software was developed with the support of "Project for advancement of software usability in materials science" of The Institute for Solid State Physics, The University of Tokyo.

The installation of moller on Miyabi and Genkai was supported by JST Moonshot R&D Program (Grant Number JPMJMS24A3).

Citation
----------------------------------------------------------------

When publishing results obtained using this software, we would appreciate it if you cite the following paper:

Kazuyoshi Yoshimi, Yuichi Motoyama, Tatsumi Aoyama, Mitsuaki Kawamura, and Naoki Kawashima,
"Project for advancement of software usability in materials science",
Science and Technology of Advanced Materials: Methods **5**, 2564055 (2025).
`https://doi.org/10.1080/27660400.2025.2564055 <https://doi.org/10.1080/27660400.2025.2564055>`_

BibTeX entry:

.. code-block:: bibtex

@article{Yoshimi2025,
author = {Kazuyoshi Yoshimi and Yuichi Motoyama and Tatsumi Aoyama and Mitsuaki Kawamura and Naoki Kawashima},
title = {Project for advancement of software usability in materials science},
journal = {Science and Technology of Advanced Materials: Methods},
volume = {5},
number = {1},
pages = {2564055},
year = {2025},
doi = {10.1080/27660400.2025.2564055},
url = {https://doi.org/10.1080/27660400.2025.2564055}
}

Operating environment
----------------------------------------------------------------

moller was tested on the following platforms:

- Ubuntu Linux + python3

moller is available on the following supercomputer systems.
For instructions on using moller on each system, please refer to its user guide.

.. list-table::
:header-rows: 1
:widths: 20 50 30

* - System
- Operated by
- Status
* - ohtaka, kugui
- `Supercomputer Center, ISSP <https://mdcl.issp.u-tokyo.ac.jp/scc/en/>`_, The University of Tokyo
- pre-installed
* - `AOBA <https://www.cc.tohoku.ac.jp/english/>`_
- Cyberscience Center, Tohoku University
- pre-installed
* - `Genkai <https://www.cc.kyushu-u.ac.jp/scp/en/system/genkai/>`_
- Research Institute for Information Technology, Kyushu University
- pre-installed
* - `Miyabi <https://www.cc.u-tokyo.ac.jp/en/supercomputer/miyabi/service/>`_
- Information Technology Center, The University of Tokyo
- supported; pre-installation in progress (as of Aug 14, 2026)

34 changes: 17 additions & 17 deletions docs/en/source/moller/filespec/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ A job description file contains configurations to generate a batch job script by

3. prologue and epilogue sections: specifies initial settings and finalization within the batch job.

4. jobs section: specifies tasks to be carried out in the betch job script.
4. jobs section: specifies tasks to be carried out in the batch job script.

General settings
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -36,7 +36,7 @@ platform
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``system``

specifies the target system. At present, either ``ohtaka`` or ``kugui`` is accepted.
specifies the target system. At present, ``ohtaka``, ``kugui``, ``pbs`` (generic PBS-based clusters), and ``default`` (environments without a job scheduler) are accepted.

``queue``

Expand All @@ -63,18 +63,18 @@ platform
.. code-block:: yaml

options: |
--mail-type=BEGIN,END,FAIL
--mail-user=user@sample.com
--requeue
--mail-type=BEGIN,END,FAIL
--mail-user=user@sample.com
--requeue

- an example of PBS job script in the list format:

.. code-block:: yaml

options:
- -m bea
- -M user@sample.com
- -r y
- -m bea
- -M user@sample.com
- -r y

prologue, epilogue
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -100,31 +100,31 @@ jobs

``description``

provides the description of the task. It is regarded as comments.
provides the description of the task. It is regarded as comments.

``node``

specifies the degree of parallelism in one of the following formats.
specifies the degree of parallelism in one of the following formats.

- ``[`` number of processes, number of threads per process ``]``
- ``[`` number of nodes, number of processes, number of threads per process ``]``
- number of nodes

When the number of nodes is specified, the specified number of nodes are exclusively assigned to a job. Otherwise, if the required number of cores for a job is smaller than the number of cores in a node, more than one job may be allocated in a single node. If a job uses more than one node, the required number of nodes are exclusively assigned.
When the number of nodes is specified, the specified number of nodes are exclusively assigned to a job. Otherwise, if the required number of cores for a job is smaller than the number of cores in a node, more than one job may be allocated in a single node. If a job uses more than one node, the required number of nodes are exclusively assigned.

``parallel``

This parameter is set to ``true`` if the tasks of different jobs are executed in parallel. It is set to ``false`` if they are executed sequentially. The default value is ``true``.
This parameter is set to ``true`` if the tasks of different jobs are executed in parallel. It is set to ``false`` if they are executed sequentially. The default value is ``true``.

``run``

The content of the task is described in the form of shell script. The executions of MPI parallel programs or MPI/OpenMPI hybrid parallel programs are specified by
The content of the task is described in the form of shell script. The executions of MPI parallel programs or MPI/OpenMPI hybrid parallel programs are specified by

.. code-block:: bash

srun prog [arg1, ...]
where, in addition to the keyword ``srun``, ``mpirun`` or ``mpiexec`` is accepted. In the resulting job script, they are replaced by the command (e.g. ``srun`` or ``mpirun``) and the degree of parallelism specified by ``node`` parameter.

where, in addition to the keyword ``srun``, ``mpirun`` or ``mpiexec`` is accepted. In the resulting job script, they are replaced by the command (e.g. ``srun`` or ``mpirun``) and the degree of parallelism specified by ``node`` parameter.

List file
----------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/en/source/moller/tutorial/basic.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ Run batch job
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The batch job is to be submitted to the job scheduler with the batch job script.
In this example, the job script and the input parameter files are copied into the ``output`` directory, and the current directory is changed to ``output` as follows:
In this example, the job script and the input parameter files are copied into the ``output`` directory, and the current directory is changed to ``output`` as follows:

.. code-block:: bash

Expand Down
Binary file added docs/images/moller_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading