script_submitter v3.6: runs never finish when the script runs without a scheduler

[RicardoMtzC]

Summary

script_submitter v3.6's stream_output job tails the script output with tail -f, which never exits on its own. Every submit path is meant to end with a Cancel Streaming step (uses: parallelworks/cancel-jobs → stream_output). Three paths are missing it, so the subworkflow never reaches a terminal state even after the submitted script has finished.

Which paths

variant	ssh_job	pbs_job	slurm_job	scheduler_agent_job
general.yaml	missing	✅	✅	✅
emed.yaml	missing	✅	✅	✅
hsp.yaml	✅	✅	✅	✅
noaa.yaml	✅	✅	✅	missing

hsp.yaml is the only variant where all four paths cancel, so it doubles as the reference for what ssh_job should look like: Create Script | Submit Script | Wait for Script | Cancel Streaming.

This looks like a copy-paste omission during the consolidation — v3.5 had 4 cancel sites, v3.6's general.yaml has 3, and the step v3.5's ssh_job ended with (Cancel Streaming) was replaced by Wait for Script without the cancel being re-added below it.

Evidence

Observed on a live AWS-Slurm cluster on canary (v7.100.0), composing workflows/script_submitter/v3.6/general.yaml from a matrix job with scheduler: false:

• ssh_job → completed, all four steps (Create Script, Submit Script, Wait for Script, POST Submit Script)
• stream_output → running, stuck on Stream Output
• The run was still running 20+ minutes later. The worker script body was literally date.
• Exactly 3 of 20 matrix jobs hung, matching n_workers: 3 — the 17 skipped slots finished fine.

The identical composition against v3.5 (which had Cancel Streaming in ssh_job) completes in 29.6s.

Impact

Any workflow composing v3.6 that runs without a scheduler (→ ssh_job on general/emed), or uses the scheduler agent on noaa, with submit_and_exit false or unset, hangs forever instead of completing.

The session/endpoint workflows in this repo don't surface it — they're long-lived and get torn down, so nothing ever waits for a terminal state. It only bites batch-style compositions that have to reach completed. Found while repointing the ACTIVATE e2e parallel-workers test at this repo (parallelworks/core#19639), which is blocked on it.

Possible fixes — your call

1. Re-add Cancel Streaming to the three paths. Verbatim from the paths that already have it, keeping the if: ${{ inputs.submit_and_exit != true }} guard. Smallest change, makes all 16 path/variant combinations consistent. 18 lines:

--- a/workflows/script_submitter/v3.6/general.yaml
+++ b/workflows/script_submitter/v3.6/general.yaml
@@ -175,6 +175,12 @@ jobs:
             sleep 15
           done
           echo "::notice::Process ${pid} finished"
+      - name: Cancel Streaming
+        if: ${{ inputs.submit_and_exit != true }}
+        uses: parallelworks/cancel-jobs
+        with:
+          jobs:
+            - stream_output

…the same after Wait for Script in emed.yaml, and after Submit Script and Exit in noaa.yaml's scheduler_agent_job. I have this ready as a branch/patch and can open a PR if someone grants push (or I can attach the patch here).

2. Make stream_output self-terminating — e.g. tail --pid=<submitted pid> or a sentinel written by the submit step — so no path needs an explicit cancel. Larger change, but removes the whole class of bug rather than the three instances, and is robust to new submit paths being added later.

Related question if you go with (1): the per-path Cancel Streaming block is now duplicated 16 times across 4 variants. Worth factoring out, or is the duplication deliberate given marketplace registrations pin these paths?

Repro

# From a workflow with a compute-clusters input and scheduler=false:
#   uses: github/parallelworks/workflows@canary
#   with:
#     $yaml: workflows/script_submitter/v3.6/general.yaml
#     use_existing_script: true
#     script_path: <a script that exits immediately, e.g. `date`>
#     scheduler: false
#     use_scheduler_agent: false
# -> ssh_job completes, stream_output stays running, run never terminates.