Carry out the session control directives PRRTE can honor

[rhc54]

Fixes #2589

The problem

PMIx_Session_control is the API a scheduler uses to create a session, operate on
it while it runs, and reclaim it. PRRTE accepted the call and did almost none of
it: every operational directive returned PMIX_ERR_NOT_SUPPORTED, and an
instantiation built a session and a job object but never launched anything.

So a scheduler driving PRRTE had no way to start work in a session it had just
created, and no way to pause, signal, extend or terminate one afterwards. The
placeholder in docs/how-things-work/schedulers/flow_of_control.rst has been
waiting for this.

What this does

Implements the whole surface. Instantiation gives the session the nodes it was
described with, extends the DVM across any that are new, and launches the
applications it was given into it — answering with the session id, the allocation
id, and the launched namespace. Pause and resume stop and continue every job in
the session; preempt and restore do the same for the jobs a PMIX_NSPACE
qualifier names; signal delivers an arbitrary signal; extend adds nodes or revises
the time limit; terminate kills the jobs and reclaims the session once they have
retired. A session id of UINT32_MAX addresses every session the caller controls.

A scheduler may impose a lifetime on a session (PMIX_ALLOC_TIME/PMIX_TIMEOUT);
when it expires the session is terminated exactly as an explicit request would.
What becomes of a reclaimed session's nodes follows its PMIX_ALLOC_INHERITANCE
disposition, defaulting to the DVM's general pool — handing them back to the
scheduler is the destructive choice and has to be asked for.

The user-facing description of every directive, including where PRRTE deviates and
why, is the new docs/how-things-work/schedulers/session_control.rst.

Two deliberate deviations

Preemption is suspension-based. The attribute describes preemption as halting
the jobs and recovering their resources. Recovering them means checkpointing the
jobs out and restoring them later, which PRRTE cannot do — and terminating them
instead would make the paired PMIX_SESSION_RESTORE impossible to honor at all.
So a preempted job stops and holds its resources, and a restored job resumes where
it was. A scheduler that genuinely needs the nodes back should terminate the
session.

PMIX_SESSION_COMPLETE is outbound only. It is what an RTE says to its
scheduler, not the other way round: PRRTE is the only party that knows when the
jobs have finished and the nodes are free. Inbound it is refused; PRRTE now sends
it — carrying each job's termination status — when a session the scheduler
instantiated is reclaimed.

Two design points worth review

A session created through this API is detached by default, which is the
opposite of an allocation reservation. A reservation is nodes a tool took for its
own use and dies with that tool. A session is named, addressable by anyone
authorized, and destroyed by an explicit terminate, by its own time limit, or —
when it was created to run apps — when those apps finish. Following the requester
instead makes the API unusable for what it exists to do: the requester is a tool,
its namespace lasts only as long as the request, and the session was being
reclaimed out from under its own jobs the moment the caller disconnected (this is
not hypothetical — it is what the multi-node harness caught first).
PMIX_SESSION_SEP given as false opts back into requester lifetime.

A DVM with no scheduler answers for itself. The scheduler owns sessions when
there is one, but refusing session control outright when there is not left the
capability unreachable on an ordinary prte DVM — and impossible to test. Requests
are gated on the same ownership check that governs spawning into a reservation, so
a caller can only act on an allocation it owns. The relay from a non-master daemon
makes the same decision, so a caller gets the same answer whichever daemon it
happens to be attached to.

Pre-existing bugs this uncovered

Each is a separate commit ahead of the feature, so they can be reviewed and
bisected on their own. The middle two were only ever reachable from a non-master
daemon, which is why they had survived: both APIs are normally driven from the
head node, where the master answers directly and the relay never runs.

1. prte_plm_base_prted_signal_local_procs() packed the wrong bytes as the
   target namespace. A pmix_nspace_t is an array type, so as a parameter it
   has decayed to a char * — and &job is the address of that pointer variable,
   not of the name. PMIX_PROC_NSPACE copies PMIX_MAX_NSLEN+1 bytes from
   whatever it is handed, so a stack fragment went on the wire, every daemon
   matched no job, and the signal was silently dropped. Every signal delivered
   through prte_plm.signal_job() was lost; PMIx_Job_control signals were
   unaffected (that path uses a real local array).

2. pmix_server_alloc_request_resp() answered a relayed request into the void.
   It passed the request tracker as the callback data where the original caller's
   cbdata belongs. PMIx uses that value to find the operation it is completing,
   so it was handed a pointer it had never issued: the client sat in its PMIx call
   until it timed out, or the library faulted on the unrecognized pointer and took
   the daemon with it. This affected every allocation request as well as every
   session-control request issued from a non-master daemon.

3. prte_pmix_set_scheduler() re-probed on every request.
   PMIx_tool_attach_to_server is a blocking call that hunts the filesystem for a
   rendezvous and tries to connect to what it finds — and it ran on the PRRTE
   progress thread, the only thread driving the DVM, in front of every allocation
   and every session-control request. On a DVM with no scheduler (the ordinary case,
   where it can only ever fail) that is pure cost on the one thread that must not be
   spent. It now looks once; a scheduler that appears later announces itself by
   attaching to us.

4. prte_ras_base_teardown_reservation() could shrink the master out of its own
   DVM. When handing a reservation's nodes back to the scheduler it collected the
   daemon rank of every member node, with nothing excluding our own. A reservation
   may legitimately contain the head node, and ordering our own daemon to depart
   terminates the DVM, taking every other job with it.

Testing

• Unit — new coverage for the session time-limit parser in
  test/unit/prted/. (Writing it caught that PMIx_Argv_split discards empty
  tokens, so ":30" and "1::2" were being silently reinterpreted as valid
  durations rather than refused.)
• make check — all suites pass.
• Offline mapper harness — 1180/1180.
• Docs — build warning-free; new session_control.rst wired into the
  scheduler-integration section.
• Single node — examples/sessionctrl.c drives the full lifecycle:
  instantiate + launch, pause (procs observably in T), resume, preempt/restore,
  signal, time-limit expiry, terminate, and the error cases.
• Multi node — new test_session phase in contrib/dockerswarm
  (14 cases): a reservation actually withholding its nodes from a general job, a
  request relayed from a non-master daemon, both nodes' procs stopping and
  resuming, a signal reaching the session's jobs on every node they occupy, a
  session created to run apps being reclaimed when they end, and a standing
  reservation persisting until explicitly terminated.
  Full suite: 395 passed, 0 failed, 2 skipped (both skips pre-existing PMIx
  capability gates, unrelated).

The multi-node run is what found items 1–3 above, plus a self-deadlock of my own
making: _send_alloc_resp releases the request twice — explicitly, and again
through its caddy's destructor — so a new caller has to hand it over holding the
extra PMIX_RETAIN the allocation branch already takes, or the last release runs
the destructor while the object lock is held. That path also has to be
thread-shifted out of the RML receive before it drives the PLM, as the allocation
branch does.

🤖 Generated with Claude Code