DeveloperGuide.md

Developer Guide: Creating a New Workflow

A session workflow lives in one directory, workflows/my-session/, and needs three things:

File	Runs on	Purpose
app/controller.sh	Controller (login) node	Install software, download dependencies
app/start-template.sh	Controller or compute node	Start the web service
yamls/<variant>.yaml (e.g. yamls/general.yaml)	Platform	Define the UI form, generate inputs.sh, orchestrate

app/ holds everything the run needs (scripts + support files) and is the only subtree the workflow sparse-checkouts; yamls/, thumbnails/, README, and build tooling stay outside it.

The controller node always has internet access. The compute node may not.

The platform docs are the source of truth — consult them when anything here is unclear or behaves differently: building workflows (YAML fields · inputs & expressions · actions) · endpoint sessions · pw endpoints CLI.

Working with an AI assistant? The repo ships a Claude Code skill — .claude/skills/activate-workflows/ — that encodes this process plus the platform reference. In Claude Code: "Using the activate-workflows skill, create a new interactive session workflow for [deployment] that [does X]."

1. The controller script

workflows/my-session/app/controller.sh runs before the service starts, on the login node. Use it for anything needing internet. All inputs.sh variables are available.

#!/usr/bin/env bash
set -o pipefail

if [ -z ${service_parent_install_dir} ]; then
    service_parent_install_dir=${HOME}/pw/software
fi

if ! [ -f "${service_parent_install_dir}/my-server" ]; then
    echo "Installing my-server..."
    mkdir -p ${service_parent_install_dir}
    wget https://example.com/my-server.tar.gz -O /tmp/my-server.tar.gz
    tar -xzf /tmp/my-server.tar.gz -C ${service_parent_install_dir}
fi

Keep it idempotent — check whether software exists before installing.

2. The start script

workflows/my-session/app/start-template.sh starts the web service. The platform provides service_port — your service must listen on it. All inputs.sh variables are available.

#!/bin/bash
if [ -z ${service_parent_install_dir} ]; then
    service_parent_install_dir=${HOME}/pw/software
fi

# cancel.sh lets the platform stop the service
echo '#!/bin/bash' > cancel.sh
chmod +x cancel.sh

${service_parent_install_dir}/my-server --port=${service_port} &
pid=$!
echo "kill ${pid}" >> cancel.sh

sleep inf

Requirements: listen on service_port, write a cancel.sh, end with sleep inf (or run the service in the foreground).

3. The workflow YAML

workflows/my-session/yamls/general.yaml. Its jobs:

1. preprocessing — parallelworks/checkout of this repo (sparse: workflows/my-session/app, plus tools/... if the scripts use the shared tools), generate inputs.sh from the form values + PW_* environment, run inputs.sh + controller.sh inline, and assemble the start script (inputs.sh + a cleanup trap + start-template.sh).
2. session_runner (the job name kept for history) — submit the start script via workflows/script_submitter/v3.6/<variant>.yaml (uses: github/parallelworks/workflows@canary).
3. wait_for_endpoint — poll pw endpoints list until the endpoint named <service.name>-${PW_RUN_SLUG} is online, then leave the service running (SKIP_CLEANUP marker) and cancel the submitter's wait.

Copy a real one instead of writing from scratch — workflows/webshell/yamls/general.yaml is the smallest complete example; workflows/jupyterlab/yamls/general.yaml shows a conda install plus support files; workflows/streamlit/yamls/general.yaml + its scripts show a Singularity/SIF service (SIF pulled via oras, .def + build-container.sh alongside). Key parts to adapt:

• the hidden service.name input (endpoint name prefix),
• the sparse-checkout paths (workflows/my-session/app, tools/...),
• the cat workflows/my-session/app/controller.sh / start-template.sh lines,
• the service input group (your form fields → inputs.sh variables).

4. Platform variants

One YAML per deployment: yamls/general.yaml (standard SLURM/PBS clusters), plus emed.yaml / hsp.yaml / noaa.yaml where the workflow is offered there. Variants differ in scheduler directives, partitions, module loads, and defaults — copy the matching variant of a similar workflow (they pass their variant's workflows/script_submitter/v3.6/<variant>.yaml).

5. Testing

Every workflow is tested end-to-end at least once, and any end-to-end test is recorded under workflows/my-session/tests/<variant>/. The test layout, the test file keys, the pass criteria and the result columns are documented once, in tools/tests/README.md. Start from an existing test, e.g. workflows/webshell/tests/<variant>/<test-name>.json.

Push first — the YAML pulls this repo from GitHub at run time, so local edits to app/ are invisible until they are on the referenced branch. Then:

python3 tools/tests/run-workflow-test.py workflows/my-session/tests/<variant>/<test-name>.json

Commit the test and the rows the runner appends to its CSV with your change; never edit a CSV by hand.

Verify cleanup on cancel — part of testing, every time. Cancel a run mid-flight (pw workflows runs cancel <slug> while the service is starting or serving) and confirm the cleanup actually ran: no service processes left (ps -x), no scheduler job (squeue/qstat when scheduler:true), no container instances (singularity instance list, docker ps), no stray listeners. Apps that daemonize and re-parent to PID 1 (e.g. RStudio's rsession) can survive the tree kill and need handling in cancel.sh. Write cancel.sh at the very top of the start script so a cancel at any moment finds it.

While iterating, point the YAML's checkout branch: at a development branch and restore it to canary before the PR merges (canary only accepts pull requests).

6. Debugging

Everything a run did is in its job dir on the execution node: ~/pw/jobs/<run-slug>/ for CLI file runs, or ~/pw/jobs/<workflow-name>/<run-number, 5 digits>/ for registered workflows.

• run.<JOBID>.out — the service's stdout/stderr
• logs/<job>/step_N/step.out, step.exit — per-step trace and exit code
• logs/<job>/step_N/script-unstable.sh — the rendered step: every ${{ input }} appears as the literal value the form sent. When a value seems ignored (a default not applied, an empty field), read this first.
• From any machine: pw workflows runs errors <slug> and pw workflows runs logs <slug>.

Common pitfalls

• Testing unpushed code — the checkout fetches GitHub, not your working tree.
• Relative YAML path in pw workflows run — parsed as a git host; use absolute.
• Composing checkout paths wrong — checked-out files materialize at ${PW_PARENT_JOB_DIR}/workflows/<name>/app/… (or …/<impl>/…), including paths built from variables ("${PW_PARENT_JOB_DIR}/${service_name}"-style bugs surface only at run time).
• Globs copy everything in the directory they target. Some preprocessing steps collect support files with a glob — e.g. jupyter runs cp workflows/jupyter/app/*.yaml . to gather its conda-env files. That only works because app/ contains nothing but runtime files: the workflow's own variant YAMLs live in yamls/, outside app/. Keep it that way — a workflow YAML (or any unrelated file) placed inside app/ would be swept into every run.
• Single-attempt ghcr pulls — ghcr intermittently rate-limits anonymous pulls; use tools/oras/libs.sh:oras_pull_file (it retries) and keep packages public.
• A registered workflow ignoring your defaults — the registration pins one YAML path (pw workflows get <name> → remote.yaml); if it points at the wrong variant, the form (and its defaults) are the wrong variant's.
• "Authentication has expired" — pw tokens lapse; re-run pw auth.

Appendix: converting a legacy workflow

Nothing in this repo uses the older session pattern (a sessions: block + the session_runner subworkflow). To convert a legacy workflow to the endpoint pattern and bring it here, follow .claude/skills/activate-workflows/references/session-to-endpoint-upgrade.md; which workflows are still legacy and where they live is in MIGRATION.md.