diff --git a/README.md b/README.md index 743cccf..018bc28 100644 --- a/README.md +++ b/README.md @@ -236,19 +236,25 @@ way Ctrl-C would, so a shell loop around it aborts. A `background` script (or `wf run -b NAME`) is detached instead: it runs under a supervisor of its own — the same process-group and cleanup -handling, minus the terminal — with its output in -`.git/workforest/logs/NAME/WORKTREE.log` of the main checkout, and `wf` -returns as soon as it is clearly running (a command that dies right away -is reported with the tail of its log). That way several long-running -scripts can be started from one terminal. `wf stop NAME` stops this -worktree's instance, foreground or background, from any terminal; -`--all` stops every worktree's. A script runs once per worktree: starting -one that is already running there fails (`wf stop` it first). - -An `exclusive` script runs at most once per project: starting it stops the -running instance in any worktree (this one included), waits for that -instance's cleanup to finish, and only then starts. Stopping — by `wf -stop` or by preemption — is SIGTERM, then SIGKILL after `stop_timeout` +handling, minus the terminal — with its output in a log file under +`.git/workforest/logs/NAME/` of the main checkout, and `wf` returns as +soon as it is clearly running (a command that dies right away is reported +with the tail of its log; the path is printed when it starts). That way +several long-running scripts can be started from one terminal. Any number +of instances of a script may run at once, in one worktree or across +several: each keeps a record and — in the background — a log of its own, +named after the worktree and the pid of the `wf run` that owns it +(`WORKTREE.PID.log`), and each runs its own `cleanup` when it ends, so a +`cleanup` has to tolerate a sibling instance still running. `wf stop +NAME` stops every instance in this worktree, foreground or background, +started from any terminal; `--all` stops every worktree's. Logs of +instances that are no longer running are removed the next time the script +starts there. Use `exclusive` for a script that must not run twice. + +An `exclusive` script runs at most once per project: starting it stops +every running instance in any worktree (this one included), waits for +their cleanup to finish, and only then starts. Stopping — by `wf stop` or +by preemption — is SIGTERM, then SIGKILL after `stop_timeout` seconds (30 by default; the top-level key changes it for every script, an entry's own `stop_timeout` for that one). The stopped instance's `wf run` reports ``killed by SIGTERM (stopped by `wf run backend` in 'feat-x')`` @@ -283,8 +289,9 @@ A group is a script like any other: `wf run -b`, `background`, member, waits for all their cleanups, then runs the group's own. A group takes no extra arguments; its `stop_timeout` defaults to the longest of its members'. Members stay individually visible: `wf stop MEMBER` stops -that member out from under a running group, and a member already running -in the worktree fails its step. +that member out from under a running group, and a member that is already +running elsewhere is started again — `exclusive` is what makes a member +stop the running one instead. A pipeline's steps run like consecutive `wf run`s, each with the terminal; a `background` member is started and left running while the @@ -382,7 +389,7 @@ directives for the shell function, `--porcelain`/`--json` listings, dumps). `list --json` describes the whole forest for programs — `main` (the main checkout, in the same `name`/`branch`/`path`/`dirty`/`running` shape as each entry of `worktrees`, `running` being the names of the scripts running -there) and the resolved `worktrees_dir` — and is what the editor extensions +there, each once however many instances of it run) and the resolved `worktrees_dir` — and is what the editor extensions read. ## JetBrains IDE plugin diff --git a/man/workforest.1 b/man/workforest.1 index 3921d01..c278e94 100644 --- a/man/workforest.1 +++ b/man/workforest.1 @@ -204,7 +204,8 @@ carry (true/false), and .B running -(the names of the scripts running there); +(the names of the scripts running there, each once however many instances +of it run); .B worktrees_dir is the resolved worktrees directory. .SS delete \fINAME\fR... @@ -259,9 +260,9 @@ command (see then runs however the command ended. Starting an .B exclusive -script first stops its running instance in any worktree of the project -and waits for that instance's cleanup; any other script refuses to start -while it is already running in this worktree (see +script first stops every running instance in the project and waits for +their cleanup; any other script simply joins the instances already +running, in this worktree or elsewhere (see .BR stop ). The exit status is the command's, or 128+\fIN\fR when it was killed by signal @@ -280,8 +281,10 @@ member's status once all have ended. .BR \-b ", " \-\-background Detach instead: the command runs under a supervisor of its own with its output in -.IR .git/workforest/logs/ SCRIPT / WORKTREE .log -of the main checkout, and +.IR .git/workforest/logs/ SCRIPT / WORKTREE . PID .log +of the main checkout \(em +.I PID +being the process that owns the instance \(em and .B workforest returns once it is clearly running (a command that dies right away is reported with the tail of its log). @@ -293,7 +296,7 @@ Must precede after it, it belongs to .IR ARGS . .SS stop \fISCRIPT\fR [\fB\-\-all\fR] -Stop the script's running instance in the current worktree \(em +Stop every running instance of the script in the current worktree \(em foreground or background, started from any terminal \(em with SIGTERM, then SIGKILL after .B stop_timeout @@ -304,7 +307,7 @@ under a running group. Fails when nothing is running. .TP .B \-\-all -Every worktree's instance of the script, not just this one's. +Every worktree's instances of the script, not just this one's. .SS tui [\fIMODE\fR] Interactive mode, driven by .BR fzf (1). diff --git a/man/workforest.5 b/man/workforest.5 index 64254a4..d594054 100644 --- a/man/workforest.5 +++ b/man/workforest.5 @@ -191,8 +191,9 @@ Members are named, never written inline, and keep their own settings and records: a group may name another group (a cycle is a configuration error), .B workforest stop -on a member stops it out from under a running group, and a member -already running in the worktree fails its step. +on a member stops it out from under a running group, and a member that is +already running elsewhere is started again unless it is +.BR exclusive . .TP .B background Detach by default, as @@ -211,19 +212,23 @@ variables. Its failure is a warning. .TP .B exclusive -At most one instance per project: starting the script stops the running +At most one instance per project: starting the script stops every running instance in any worktree (this one included) with SIGTERM, then SIGKILL after .B stop_timeout -seconds, waits for that instance's cleanup to finish, and only then -starts. +seconds, waits for their cleanup to finish, and only then starts. +Without it any number of instances may run at once, in one worktree or +across several, each with a record, a log, and a +.B cleanup +of its own. The stopped instance's .B workforest run reports who preempted it and exits 143. Running instances are recorded under .I .git/workforest/running/ -in the main checkout; a record is verified (pid, process group, boot) -before anything is signalled, so a crash, a reboot, or a killed +in the main checkout, one file per instance; a record is verified (pid, +process group, boot) before anything is signalled, so a crash, a reboot, +or a killed .B workforest leaves nothing to tidy by hand. A command still running after its diff --git a/src/workforest/cli.py b/src/workforest/cli.py index 126b245..fa39991 100644 --- a/src/workforest/cli.py +++ b/src/workforest/cli.py @@ -22,7 +22,7 @@ "delete": "delete worktree(s)", "checkout": "delete a worktree and check its branch out in main", "run": "run a named script from the merged config", - "stop": "stop a running script (this worktree's instance, or --all)", + "stop": "stop a running script (this worktree's instances, or --all)", "tui": "interactive mode (requires fzf)", "init": "write a commented .workforest.yaml starter", "config": "show the merged configuration and its sources", @@ -216,7 +216,7 @@ def opener_args(p: argparse.ArgumentParser) -> None: p = sub.add_parser("stop", help=SUBCOMMAND_HELP["stop"]) p.add_argument("script", metavar="SCRIPT") p.add_argument( - "--all", action="store_true", help="every worktree's instance, not just this one's" + "--all", action="store_true", help="every worktree's instances, not just this one's" ) p.set_defaults(func=_handle_stop) diff --git a/src/workforest/config.py b/src/workforest/config.py index 77917cd..3719855 100644 --- a/src/workforest/config.py +++ b/src/workforest/config.py @@ -55,7 +55,7 @@ class ScriptSpec: bulk: tuple[str, ...] | None = None # members run at once; done when all are pipeline: tuple[str, ...] | None = None # members run in order; stops at the first failure background: bool = False # detach, with output to a log file, instead of holding the terminal - exclusive: bool = False # starting it stops the running instance anywhere in the project + exclusive: bool = False # starting it stops every running instance in the project cleanup: str | None = None # runs after the command ends, however it ended stop_timeout: float | None = None # seconds between SIGTERM and SIGKILL; None: the global one diff --git a/src/workforest/examples/.workforest.yaml b/src/workforest/examples/.workforest.yaml index 6fdfa39..9859874 100644 --- a/src/workforest/examples/.workforest.yaml +++ b/src/workforest/examples/.workforest.yaml @@ -67,7 +67,8 @@ setup_scripts: # background: true detached, output in a log file (`workforest run -b` does # the same once); `workforest stop NAME` ends it # exclusive: true at most one instance per project — starting it stops -# the running one in any worktree (cleanup included) +# the running ones in any worktree (cleanup included); +# without it a script may run several times at once # cleanup: CMD runs after the command ends, however it ended # stop_timeout: N seconds between SIGTERM and SIGKILL when stopped # (default: the top-level `stop_timeout`, 30; for a diff --git a/src/workforest/hooks.py b/src/workforest/hooks.py index daf1b9c..4fc0a3c 100644 --- a/src/workforest/hooks.py +++ b/src/workforest/hooks.py @@ -257,9 +257,18 @@ class _Job: snippet: str cwd: Path env: dict[str, str] - record_path: Path + common_dir: Path # where the record and the log live tty: bool = True + def record_path(self, pid: int) -> Path: + return jobs.record_path(self.common_dir, self.name, self.cwd, pid) + + def log_path(self, pid: int) -> Path: + """The log of the instance owned by `pid`: the detached supervisor, + which names its own log from the inside and is named from the + outside by the `wf run` that forked it.""" + return jobs.log_path(self.common_dir, self.name, self.cwd, pid) + def _fileno(stream: int | IO[bytes]) -> int: return stream if isinstance(stream, int) else stream.fileno() @@ -310,14 +319,14 @@ def _spawn(job: _Job, sink: _Sink, tty_fd: int | None) -> int: return pid -def _run_command(job: _Job) -> _JobResult: +def _run_command(job: _Job, record_path: Path) -> _JobResult: """Run the job in its own process group with a job record on disk for as long as it runs.""" tty_fd = _controlling_tty() if job.tty else None with _diverted_output() as sink: pgid = _spawn(job, sink, tty_fd) jobs.write_record( - job.record_path, + record_path, jobs.JobRecord( script=job.name, worktree=str(job.cwd), @@ -329,7 +338,7 @@ def _run_command(job: _Job) -> _JobResult: ), ) code = _wait(pgid, tty_fd=tty_fd) - record = jobs.read_record(job.record_path) + record = jobs.read_record(record_path) return _JobResult(code, record.stopped_by if record else None) @@ -343,12 +352,14 @@ def _run_cleanup(spec: ScriptSpec, name: str, *, cwd: Path, env: dict[str, str]) def _run_to_completion(job: _Job) -> _JobResult: - """Command, then cleanup, then — and only then — the record goes.""" + """Command, then cleanup, then — and only then — the record goes. We + own the run, so our pid names the instance.""" + record_path = job.record_path(os.getpid()) try: - result = _run_command(job) + result = _run_command(job, record_path) _run_cleanup(job.spec, job.name, cwd=job.cwd, env=job.env) finally: - job.record_path.unlink(missing_ok=True) + record_path.unlink(missing_ok=True) return result @@ -398,7 +409,7 @@ def _exit_with(code: int) -> None: # pragma: no cover - ends a forked child os._exit(code) -def _supervise_detached(job: _Job, log_fd: int) -> None: # pragma: no cover - the forked child +def _supervise_detached(job: _Job) -> None: # pragma: no cover - the forked child """The background supervisor: our fork, in a session of its own, with the log as its stdout/stderr. Runs the command exactly like the foreground path does and exits as `wf run` would; never returns.""" @@ -408,6 +419,7 @@ def _supervise_detached(job: _Job, log_fd: int) -> None: # pragma: no cover - t devnull = os.open(os.devnull, os.O_RDONLY) os.dup2(devnull, 0) os.close(devnull) + log_fd = os.open(job.log_path(os.getpid()), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644) _redirect_output(log_fd, log_fd) os.close(log_fd) result = _run_to_completion(job) @@ -421,18 +433,28 @@ def _supervise_detached(job: _Job, log_fd: int) -> None: # pragma: no cover - t os._exit(code) -def _start_background(job: _Job, log_path: Path) -> None: +def _log_tail(log_path: Path, lines: int = 10) -> str: + """The last lines of a log, or "" when there is nothing to read (the + supervisor died before it could open one).""" + try: + text = log_path.read_text(errors="replace") + except OSError: + return "" + return "\n".join(text.strip().splitlines()[-lines:]) + + +def _start_background(job: _Job) -> None: """Fork a detached supervisor for the command and return once it is clearly running — a command that dies within the grace period is - reported with the tail of its log instead of failing invisibly.""" - log_path.parent.mkdir(parents=True, exist_ok=True) - log_fd = os.open(log_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644) + reported with the tail of its log instead of failing invisibly. The + supervisor writes the log; its pid is what names it.""" + job.log_path(os.getpid()).parent.mkdir(parents=True, exist_ok=True) # one dir per script sys.stdout.flush() sys.stderr.flush() pid = os.fork() if pid == 0: # pragma: no cover - the forked child - _supervise_detached(job, log_fd) - os.close(log_fd) + _supervise_detached(job) + log_path = job.log_path(pid) deadline = time.monotonic() + _GRACE_SECONDS while time.monotonic() < deadline: reaped, status = os.waitpid(pid, os.WNOHANG) @@ -441,10 +463,10 @@ def _start_background(job: _Job, log_path: Path) -> None: if code == 0: output.success(f"{job.name!r} already finished (log: {log_path})") return - tail = log_path.read_text(errors="replace").strip().splitlines()[-10:] + tail = _log_tail(log_path) message = f"script {job.name!r} exited with status {code} right after launch" if tail: - message += ":\n" + "\n".join(tail) + message += ":\n" + tail raise WorkforestError(message) time.sleep(0.02) output.success(f"started {job.name!r} in the background (pid {pid}, log: {log_path})") @@ -478,8 +500,7 @@ def _run_step(job: _Job) -> int: for a death by signal N — instead of raising.""" try: if job.spec.background: - common_dir = gitutil.git_common_dir(job.cwd) - _start_background(job, jobs.log_path(common_dir, job.name, job.cwd)) + _start_background(job) return 0 output.success(_start_message(job)) result = _run_to_completion(job) @@ -707,9 +728,9 @@ def _signal_members( for runner in runners: if runner.code is not None: continue - record = jobs.read_record(jobs.record_path(common_dir, runner.name, job.cwd)) - if record is not None: - jobs.signal_group(record.pgid, signum) + for member in jobs.jobs_for(common_dir, runner.name): + if member.record.owner_pid == runner.pid: # this runner's instance, not another + jobs.signal_group(member.record.pgid, signum) def _suspend_members(runners: list[_Runner], job: _Job) -> None: # pragma: no cover @@ -814,9 +835,8 @@ def _prepare( tty: bool = True, ) -> _Job: """Resolve a script and clear the way for it: an `exclusive` one first - stops its running instance anywhere in the project (that instance's - cleanup included); any other refuses to start while it is already - running in this worktree.""" + stops every running instance in the project (their cleanup included); + any other simply joins the instances already running here.""" spec = _resolve_script(config, name) snippet = spec.command or "" if extra_args: @@ -832,18 +852,11 @@ def _prepare( by=f"`wf run {name}` in {cwd.name!r}", env=env, ) - record_path = jobs.record_path(common_dir, name, cwd) - # One instance per script per worktree: records and logs are keyed that - # way, and `wf stop NAME` means "the" instance here. - if (existing := jobs.read_record(record_path)) is not None: - if jobs.classify(existing) is jobs.JobState.STALE: - record_path.unlink(missing_ok=True) - else: - raise WorkforestError( - f"{name!r} is already running in {cwd.name!r} (pid {existing.pgid}); " - f"`wf stop {name}` first" - ) - return _Job(config, spec, name, snippet, cwd, env, record_path, tty) + # Any number of instances may share a worktree, each recorded and + # logged under the pid of the run that owns it; what dead ones left + # behind goes now. + jobs.prune(common_dir, name, cwd) + return _Job(config, spec, name, snippet, cwd, env, common_dir, tty) def run_named_script( @@ -864,7 +877,7 @@ def run_named_script( """ job = _prepare(config, name, cwd=cwd, env=env, extra_args=extra_args) if job.spec.background if background is None else background: - _start_background(job, jobs.log_path(gitutil.git_common_dir(cwd), name, cwd)) + _start_background(job) return output.success(_start_message(job)) _raise_for(_run_to_completion(job), name) diff --git a/src/workforest/jobs.py b/src/workforest/jobs.py index c071823..e65e648 100644 --- a/src/workforest/jobs.py +++ b/src/workforest/jobs.py @@ -2,10 +2,12 @@ One JSON record per running `wf run`, under the repository's common git dir so every worktree of a project sees the same set: -`/workforest/running/