Allow concurrent script runs in the same worktree

README.md

@@ -236,19 +236,25 @@ way Ctrl-C would, so a shell loop around it aborts.
 
 A `background` script (or `wf run -b NAME`) is detached instead: it runs
 under a supervisor of its own — the same process-group and cleanup
-handling, minus the terminal — with its output in
-`.git/workforest/logs/NAME/WORKTREE.log` of the main checkout, and `wf`
-returns as soon as it is clearly running (a command that dies right away
-is reported with the tail of its log). That way several long-running
-scripts can be started from one terminal. `wf stop NAME` stops this
-worktree's instance, foreground or background, from any terminal;
-`--all` stops every worktree's. A script runs once per worktree: starting
-one that is already running there fails (`wf stop` it first).
-
-An `exclusive` script runs at most once per project: starting it stops the
-running instance in any worktree (this one included), waits for that
-instance's cleanup to finish, and only then starts. Stopping — by `wf
-stop` or by preemption — is SIGTERM, then SIGKILL after `stop_timeout`
+handling, minus the terminal — with its output in a log file under
+`.git/workforest/logs/NAME/` of the main checkout, and `wf` returns as
+soon as it is clearly running (a command that dies right away is reported
+with the tail of its log; the path is printed when it starts). That way
+several long-running scripts can be started from one terminal. Any number
+of instances of a script may run at once, in one worktree or across
+several: each keeps a record and — in the background — a log of its own,
+named after the worktree and the pid of the `wf run` that owns it
+(`WORKTREE.PID.log`), and each runs its own `cleanup` when it ends, so a
+`cleanup` has to tolerate a sibling instance still running. `wf stop
+NAME` stops every instance in this worktree, foreground or background,
+started from any terminal; `--all` stops every worktree's. Logs of
+instances that are no longer running are removed the next time the script
+starts there. Use `exclusive` for a script that must not run twice.
+
+An `exclusive` script runs at most once per project: starting it stops
+every running instance in any worktree (this one included), waits for
+their cleanup to finish, and only then starts. Stopping — by `wf stop` or
+by preemption — is SIGTERM, then SIGKILL after `stop_timeout`
 seconds (30 by default; the top-level key changes it for every script, an
 entry's own `stop_timeout` for that one). The stopped instance's `wf run`
 reports ``killed by SIGTERM (stopped by `wf run backend` in 'feat-x')``
@@ -283,8 +289,9 @@ A group is a script like any other: `wf run -b`, `background`,
 member, waits for all their cleanups, then runs the group's own. A group
 takes no extra arguments; its `stop_timeout` defaults to the longest of
 its members'. Members stay individually visible: `wf stop MEMBER` stops
-that member out from under a running group, and a member already running
-in the worktree fails its step.
+that member out from under a running group, and a member that is already
+running elsewhere is started again — `exclusive` is what makes a member
+stop the running one instead.
 
 A pipeline's steps run like consecutive `wf run`s, each with the
 terminal; a `background` member is started and left running while the
@@ -382,7 +389,7 @@ directives for the shell function, `--porcelain`/`--json` listings, dumps).
 `list --json` describes the whole forest for programs — `main` (the main
 checkout, in the same `name`/`branch`/`path`/`dirty`/`running` shape as each
 entry of `worktrees`, `running` being the names of the scripts running
-there) and the resolved `worktrees_dir` — and is what the editor extensions
+there, each once however many instances of it run) and the resolved `worktrees_dir` — and is what the editor extensions
 read.
 
 ## JetBrains IDE plugin

man/workforest.1

@@ -204,7 +204,8 @@ carry
 (true/false),
 and
 .B running
-(the names of the scripts running there);
+(the names of the scripts running there, each once however many instances
+of it run);
 .B worktrees_dir
 is the resolved worktrees directory.
 .SS delete \fINAME\fR...
@@ -259,9 +260,9 @@ command (see
 then runs however the command ended.
 Starting an
 .B exclusive
-script first stops its running instance in any worktree of the project
-and waits for that instance's cleanup; any other script refuses to start
-while it is already running in this worktree (see
+script first stops every running instance in the project and waits for
+their cleanup; any other script simply joins the instances already
+running, in this worktree or elsewhere (see
 .BR stop ).
 The exit status is the command's, or 128+\fIN\fR when it was killed by
 signal
@@ -280,8 +281,10 @@ member's status once all have ended.
 .BR \-b ", " \-\-background
 Detach instead: the command runs under a supervisor of its own with its
 output in
-.IR .git/workforest/logs/ SCRIPT / WORKTREE .log
-of the main checkout, and
+.IR .git/workforest/logs/ SCRIPT / WORKTREE . PID .log
+of the main checkout \(em
+.I PID
+being the process that owns the instance \(em and
 .B workforest
 returns once it is clearly running (a command that dies right away is
 reported with the tail of its log).
@@ -293,7 +296,7 @@ Must precede
 after it, it belongs to
 .IR ARGS .
 .SS stop \fISCRIPT\fR [\fB\-\-all\fR]
-Stop the script's running instance in the current worktree \(em
+Stop every running instance of the script in the current worktree \(em
 foreground or background, started from any terminal \(em with SIGTERM,
 then SIGKILL after
 .B stop_timeout
@@ -304,7 +307,7 @@ under a running group.
 Fails when nothing is running.
 .TP
 .B \-\-all
-Every worktree's instance of the script, not just this one's.
+Every worktree's instances of the script, not just this one's.
 .SS tui [\fIMODE\fR]
 Interactive mode, driven by
 .BR fzf (1).

man/workforest.5

@@ -191,8 +191,9 @@ Members are named, never written inline, and keep their own settings and
 records: a group may name another group (a cycle is a configuration
 error),
 .B workforest stop
-on a member stops it out from under a running group, and a member
-already running in the worktree fails its step.
+on a member stops it out from under a running group, and a member that is
+already running elsewhere is started again unless it is
+.BR exclusive .
 .TP
 .B background
 Detach by default, as
@@ -211,19 +212,23 @@ variables.
 Its failure is a warning.
 .TP
 .B exclusive
-At most one instance per project: starting the script stops the running
+At most one instance per project: starting the script stops every running
 instance in any worktree (this one included) with SIGTERM, then SIGKILL
 after
 .B stop_timeout
-seconds, waits for that instance's cleanup to finish, and only then
-starts.
+seconds, waits for their cleanup to finish, and only then starts.
+Without it any number of instances may run at once, in one worktree or
+across several, each with a record, a log, and a
+.B cleanup
+of its own.
 The stopped instance's
 .B workforest run
 reports who preempted it and exits 143.
 Running instances are recorded under
 .I .git/workforest/running/
-in the main checkout; a record is verified (pid, process group, boot)
-before anything is signalled, so a crash, a reboot, or a killed
+in the main checkout, one file per instance; a record is verified (pid,
+process group, boot) before anything is signalled, so a crash, a reboot,
+or a killed
 .B workforest
 leaves nothing to tidy by hand.
 A command still running after its

src/workforest/cli.py

@@ -22,7 +22,7 @@
     "delete": "delete worktree(s)",
     "checkout": "delete a worktree and check its branch out in main",
     "run": "run a named script from the merged config",
-    "stop": "stop a running script (this worktree's instance, or --all)",
+    "stop": "stop a running script (this worktree's instances, or --all)",
     "tui": "interactive mode (requires fzf)",
     "init": "write a commented .workforest.yaml starter",
     "config": "show the merged configuration and its sources",
@@ -216,7 +216,7 @@ def opener_args(p: argparse.ArgumentParser) -> None:
     p = sub.add_parser("stop", help=SUBCOMMAND_HELP["stop"])
     p.add_argument("script", metavar="SCRIPT")
     p.add_argument(
-        "--all", action="store_true", help="every worktree's instance, not just this one's"
+        "--all", action="store_true", help="every worktree's instances, not just this one's"
     )
     p.set_defaults(func=_handle_stop)
 

src/workforest/config.py

@@ -55,7 +55,7 @@ class ScriptSpec:
     bulk: tuple[str, ...] | None = None  # members run at once; done when all are
     pipeline: tuple[str, ...] | None = None  # members run in order; stops at the first failure
     background: bool = False  # detach, with output to a log file, instead of holding the terminal
-    exclusive: bool = False  # starting it stops the running instance anywhere in the project
+    exclusive: bool = False  # starting it stops every running instance in the project
     cleanup: str | None = None  # runs after the command ends, however it ended
     stop_timeout: float | None = None  # seconds between SIGTERM and SIGKILL; None: the global one
 

src/workforest/examples/.workforest.yaml

@@ -67,7 +67,8 @@ setup_scripts:
 #   background: true  detached, output in a log file (`workforest run -b` does
 #                     the same once); `workforest stop NAME` ends it
 #   exclusive: true   at most one instance per project — starting it stops
-#                     the running one in any worktree (cleanup included)
+#                     the running ones in any worktree (cleanup included);
+#                     without it a script may run several times at once
 #   cleanup: CMD      runs after the command ends, however it ended
 #   stop_timeout: N   seconds between SIGTERM and SIGKILL when stopped
 #                     (default: the top-level `stop_timeout`, 30; for a

src/workforest/hooks.py

@@ -257,9 +257,18 @@ class _Job:
     snippet: str
     cwd: Path
     env: dict[str, str]
-    record_path: Path
+    common_dir: Path  # where the record and the log live
     tty: bool = True
 
+    def record_path(self, pid: int) -> Path:
+        return jobs.record_path(self.common_dir, self.name, self.cwd, pid)
+
+    def log_path(self, pid: int) -> Path:
+        """The log of the instance owned by `pid`: the detached supervisor,
+        which names its own log from the inside and is named from the
+        outside by the `wf run` that forked it."""
+        return jobs.log_path(self.common_dir, self.name, self.cwd, pid)
+
 
 def _fileno(stream: int | IO[bytes]) -> int:
     return stream if isinstance(stream, int) else stream.fileno()
@@ -310,14 +319,14 @@ def _spawn(job: _Job, sink: _Sink, tty_fd: int | None) -> int:
     return pid
 
 
-def _run_command(job: _Job) -> _JobResult:
+def _run_command(job: _Job, record_path: Path) -> _JobResult:
     """Run the job in its own process group with a job record on disk for
     as long as it runs."""
     tty_fd = _controlling_tty() if job.tty else None
     with _diverted_output() as sink:
         pgid = _spawn(job, sink, tty_fd)
         jobs.write_record(
-            job.record_path,
+            record_path,
             jobs.JobRecord(
                 script=job.name,
                 worktree=str(job.cwd),
@@ -329,7 +338,7 @@ def _run_command(job: _Job) -> _JobResult:
             ),
         )
         code = _wait(pgid, tty_fd=tty_fd)
-    record = jobs.read_record(job.record_path)
+    record = jobs.read_record(record_path)
     return _JobResult(code, record.stopped_by if record else None)
 
 
@@ -343,12 +352,14 @@ def _run_cleanup(spec: ScriptSpec, name: str, *, cwd: Path, env: dict[str, str])
 
 
 def _run_to_completion(job: _Job) -> _JobResult:
-    """Command, then cleanup, then — and only then — the record goes."""
+    """Command, then cleanup, then — and only then — the record goes. We
+    own the run, so our pid names the instance."""
+    record_path = job.record_path(os.getpid())
     try:
-        result = _run_command(job)
+        result = _run_command(job, record_path)
         _run_cleanup(job.spec, job.name, cwd=job.cwd, env=job.env)
     finally:
-        job.record_path.unlink(missing_ok=True)
+        record_path.unlink(missing_ok=True)
     return result
 
 
@@ -398,7 +409,7 @@ def _exit_with(code: int) -> None:  # pragma: no cover - ends a forked child
     os._exit(code)
 
 
-def _supervise_detached(job: _Job, log_fd: int) -> None:  # pragma: no cover - the forked child
+def _supervise_detached(job: _Job) -> None:  # pragma: no cover - the forked child
     """The background supervisor: our fork, in a session of its own, with
     the log as its stdout/stderr. Runs the command exactly like the
     foreground path does and exits as `wf run` would; never returns."""
@@ -408,6 +419,7 @@ def _supervise_detached(job: _Job, log_fd: int) -> None:  # pragma: no cover - t
         devnull = os.open(os.devnull, os.O_RDONLY)
         os.dup2(devnull, 0)
         os.close(devnull)
+        log_fd = os.open(job.log_path(os.getpid()), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)
         _redirect_output(log_fd, log_fd)
         os.close(log_fd)
         result = _run_to_completion(job)
@@ -421,18 +433,28 @@ def _supervise_detached(job: _Job, log_fd: int) -> None:  # pragma: no cover - t
         os._exit(code)
 
 
-def _start_background(job: _Job, log_path: Path) -> None:
+def _log_tail(log_path: Path, lines: int = 10) -> str:
+    """The last lines of a log, or "" when there is nothing to read (the
+    supervisor died before it could open one)."""
+    try:
+        text = log_path.read_text(errors="replace")
+    except OSError:
+        return ""
+    return "\n".join(text.strip().splitlines()[-lines:])
+
+
+def _start_background(job: _Job) -> None:
     """Fork a detached supervisor for the command and return once it is
     clearly running — a command that dies within the grace period is
-    reported with the tail of its log instead of failing invisibly."""
-    log_path.parent.mkdir(parents=True, exist_ok=True)
-    log_fd = os.open(log_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)
+    reported with the tail of its log instead of failing invisibly. The
+    supervisor writes the log; its pid is what names it."""
+    job.log_path(os.getpid()).parent.mkdir(parents=True, exist_ok=True)  # one dir per script
     sys.stdout.flush()
     sys.stderr.flush()
     pid = os.fork()
     if pid == 0:  # pragma: no cover - the forked child
-        _supervise_detached(job, log_fd)
-    os.close(log_fd)
+        _supervise_detached(job)
+    log_path = job.log_path(pid)
     deadline = time.monotonic() + _GRACE_SECONDS
     while time.monotonic() < deadline:
         reaped, status = os.waitpid(pid, os.WNOHANG)
@@ -441,10 +463,10 @@ def _start_background(job: _Job, log_path: Path) -> None:
             if code == 0:
                 output.success(f"{job.name!r} already finished (log: {log_path})")
                 return
-            tail = log_path.read_text(errors="replace").strip().splitlines()[-10:]
+            tail = _log_tail(log_path)
             message = f"script {job.name!r} exited with status {code} right after launch"
             if tail:
-                message += ":\n" + "\n".join(tail)
+                message += ":\n" + tail
             raise WorkforestError(message)
         time.sleep(0.02)
     output.success(f"started {job.name!r} in the background (pid {pid}, log: {log_path})")
@@ -478,8 +500,7 @@ def _run_step(job: _Job) -> int:
     for a death by signal N — instead of raising."""
     try:
         if job.spec.background:
-            common_dir = gitutil.git_common_dir(job.cwd)
-            _start_background(job, jobs.log_path(common_dir, job.name, job.cwd))
+            _start_background(job)
             return 0
         output.success(_start_message(job))
         result = _run_to_completion(job)
@@ -707,9 +728,9 @@ def _signal_members(
     for runner in runners:
         if runner.code is not None:
             continue
-        record = jobs.read_record(jobs.record_path(common_dir, runner.name, job.cwd))
-        if record is not None:
-            jobs.signal_group(record.pgid, signum)
+        for member in jobs.jobs_for(common_dir, runner.name):
+            if member.record.owner_pid == runner.pid:  # this runner's instance, not another
+                jobs.signal_group(member.record.pgid, signum)
 
 
 def _suspend_members(runners: list[_Runner], job: _Job) -> None:  # pragma: no cover
@@ -814,9 +835,8 @@ def _prepare(
     tty: bool = True,
 ) -> _Job:
     """Resolve a script and clear the way for it: an `exclusive` one first
-    stops its running instance anywhere in the project (that instance's
-    cleanup included); any other refuses to start while it is already
-    running in this worktree."""
+    stops every running instance in the project (their cleanup included);
+    any other simply joins the instances already running here."""
     spec = _resolve_script(config, name)
     snippet = spec.command or ""
     if extra_args:
@@ -832,18 +852,11 @@ def _prepare(
             by=f"`wf run {name}` in {cwd.name!r}",
             env=env,
         )
-    record_path = jobs.record_path(common_dir, name, cwd)
-    # One instance per script per worktree: records and logs are keyed that
-    # way, and `wf stop NAME` means "the" instance here.
-    if (existing := jobs.read_record(record_path)) is not None:
-        if jobs.classify(existing) is jobs.JobState.STALE:
-            record_path.unlink(missing_ok=True)
-        else:
-            raise WorkforestError(
-                f"{name!r} is already running in {cwd.name!r} (pid {existing.pgid}); "
-                f"`wf stop {name}` first"
-            )
-    return _Job(config, spec, name, snippet, cwd, env, record_path, tty)
+    # Any number of instances may share a worktree, each recorded and
+    # logged under the pid of the run that owns it; what dead ones left
+    # behind goes now.
+    jobs.prune(common_dir, name, cwd)
+    return _Job(config, spec, name, snippet, cwd, env, common_dir, tty)
 
 
 def run_named_script(
@@ -864,7 +877,7 @@ def run_named_script(
     """
     job = _prepare(config, name, cwd=cwd, env=env, extra_args=extra_args)
     if job.spec.background if background is None else background:
-        _start_background(job, jobs.log_path(gitutil.git_common_dir(cwd), name, cwd))
+        _start_background(job)
         return
     output.success(_start_message(job))
     _raise_for(_run_to_completion(job), name)