Remaining run-scoped 'pw workflows runs cancel' false positives: rag-service, kasmvnc-singularity, hermes-agent

[mattshax]

Follow-up to #1058, which fixed this in the 13 templates whose call site was a single identical block. Three templates were deliberately left out because their call sites are mixed — some are legitimate, some are the bug — and they want individual review rather than a mechanical patch.

The bug being tracked

Session start templates end with a blocking pw endpoints command and then:

if [ $? -ne 0 ]; then
    echo "::error title=Error::pw endpoints command failed"
    # Fail loud: without this, wait_for_endpoint polls forever for an endpoint
    # that will never register
    pw workflows runs cancel ${PW_RUN_SLUG}
    exit 1
fi

pw endpoints run returns non-zero both when the service never started and when the workflow cancels this job after a successful launch — which is how wait_for_endpoint releases a run whose serving job would otherwise block for the full walltime. So a healthy launch cancels its own run from inside the compute job.

Observed on ollama_gguf: runs 00024/00025/00026 finished canceled with the service up; 00027/00028 finished completed after #1055 commented out that one line for ollama-gguf-container.

Sites that are this bug (4)

Each is the # Fail loud: without this, wait_for_endpoint polls forever block:

• rag-service/start-template-v4.sh:142
• kasmvnc-singularity/start-template-v4.sh:493
• hermes-agent/start-template-v4.sh:106
• hermes-agent/start-template-v4.sh:150

Fix as in #1058 — keep the fail-loud path for a launch that never registered, skip it when one did:

served_name=$(printf '%s' "${pw_endpoints_args}" | sed -n 's/.*--name[ =]\{1,\}\([^ ]*\).*/\1/p')
if [ -n "${served_name}" ] && pw endpoints list 2>/dev/null | awk '{print $1}' | grep -qxF "${served_name}"; then
    echo "::notice::Endpoint ${served_name} served until this job was cancelled; exiting cleanly"
    exit 0
fi

Note this guard is still unproven in practice — on the validation run (mp-ollama-gguf-00029) the serving job stayed alive, so the branch never executed. There is a plausible race: cancelling the job tears down the endpoint, so pw endpoints list may no longer show it by the time the check runs. A more robust signal is the SKIP_CLEANUP file, which wait_for_endpoint touches immediately before cancelling and which the job dir already carries — worth considering instead of, or as a fallback to, the endpoint lookup.

Sites that are NOT this bug — leave them

These fire before any endpoint exists, so cancelling the run is the right call:

• rag-service:57 — singularity/apptainer not found
• rag-service:64 — missing container image
• rag-service:128 — indexer died at startup

One site that wants a decision

kasmvnc-singularity/start-template-v4.sh:514, unconditional at end of script:

# No-op if the run already completed (endpoint was up); cancels it if the
# endpoint client died before ever registering.
pw workflows runs cancel ${PW_RUN_SLUG} || true

The "no-op if the run already completed" assumption should be verified rather than trusted — if a run can still be cancelled after completing, this cancels healthy kasmvnc runs the same way.

Also worth noting: not every workflow YAML has the job-scoped guard

The reason #1058 kept the cancel for genuine failures rather than deleting it is that only some YAMLs carry the Cancel Wait for Endpoint / Exit Workflow with Error steps that would otherwise stop wait_for_endpoint polling forever. Spot check: openvscode 5/10, webshell 2/5, jupyterlab-host 4/9. Adding those steps consistently would let the in-script run-scoped cancel be removed everywhere, which is the cleaner end state.

Separate, not a workflow-repo issue

The rendered workflow log for mp-ollama-gguf-00029 shows ::error title=Error::pw endpoints command failed and + pw workflows runs cancel mp-ollama-gguf-00029 under stream_output, but the job's own output file contains neither (0 occurrences of each), and the line is commented in the shipped script. That looks like stale content in the log rendering and belongs to the platform, not here.

https://claude.ai/code/session_01LQC3NvTRhZYTv83ZMEV751