Skip to content

Place each app of a job by the directive it was given - #2594

Merged
rhc54 merged 1 commit into
openpmix:masterfrom
rhc54:topic/per-app-mapping
Aug 1, 2026
Merged

Place each app of a job by the directive it was given#2594
rhc54 merged 1 commit into
openpmix:masterfrom
rhc54:topic/per-app-mapping

Conversation

@rhc54

@rhc54 rhc54 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #2563.

The problem

A placement directive written on one app of an MPMD command line was applied
to every app of the job. Asking for one app to be mapped differently
silently mapped them all that way, and there was no way to say what was
plainly meant.

The rule

The first app segment is where a command line speaks for the job. A
directive written there and nowhere else describes the whole job, however
many apps follow — that is what a single-app line has, and it is where a
qualifier spanning the job can still be written. Written on any later app it
describes that app alone; its silent siblings take the defaults, because an
app that says nothing is not agreeing with one that did.

# both apps mapped by core - the only directive is on the first app
prun --map-by core -n 4 app1 : -n 2 app2

# only app2 is mapped by node; app1 takes the default mapping
prun -n 4 app1 : --map-by node -n 2 app2

What #2563 turned out to require

The issue observed that req_mapper/last_mapper are job-level while
mapping is per-app, and predicted it "will break first when per-app mapper
selection is made to work properly." It does, and the reason is that every
mapper decided whether a job was its own by asking the job for the
policy. The job's policy is whatever default was resolved for the apps that
gave no directive, so seq, rank_file and ppr all deferred and a
per-app request for any of them was quietly placed by round_robin instead.

Gates now read the resolved options the base hands them, which in per-app
dispatch is each app's own answer. Two apps of one job can therefore be
placed by two different components.

Three things fell out of that:

  • A per-app ppr pattern kept only its count, so an app asking for two
    per package got two per whatever object the job had resolved. It now
    carries the whole N:object pattern.
  • A per-app --map-by rankfile was refused outright by the per-app parser.
    It is now accepted.
  • The mappers that number their own procs — rank_file, seq, lsf — each
    restarted from rank zero, so two apps were given the same ranks and the
    second app's procs replaced the first's in jdata->procs
    . They now take
    the base's rank cursor.

Mapper selection is the policy

req_mapper and last_mapper are removed from prte_job_map_t, and with
them from the launch message — mapping happens only on the HNP, so both
strings were shipped to every daemon on every launch and read by nothing at
the far end.

PMIX_MAPPER is refused with PMIX_ERR_NOT_SUPPORTED and is no longer
advertised in the spawn attribute lists or in PMIX_QUERY_SPAWN_SUPPORT.
Naming a component says nothing PMIX_MAPBY has not already said, and the
two can contradict each other with no answer that could be right.

An rmaps MCA parameter naming one component cannot be refused — which
components load is settled when the framework is opened, before any job
exists. A restriction that cannot serve the job's policy now fails the job
with a message naming what is loaded and what was asked for:

The requested mapping policy cannot be performed by any of the mapping
components that are loaded:

  Mapping policy:      BYCORE
  Loaded mapper(s):    seq

Which component placed an app is still recorded — on the app, only on the
HNP — and --display devel-map reports it per app.

Note on the wire format

prte_map_pack/prte_map_unpack change together in this commit. That is
only safe because mixed-version DVMs are forbidden, which was nowhere
written down; it is now, in docs/versions.rst and the developer guides.

Two unrelated fixes ride along

  • The dockerswarm harness asserted output relayed back to a tool without
    checking PMIX_CAP_IOF_DELIVER_LOCAL. The relay is compiled out without
    it, so those cases could only ever fail against an older PMIx — while the
    two beside them passed for the wrong reason.
  • The mixed-version prohibition above.

Testing

  • Warning-free --enable-debug build; lsf compiled via
    --enable-testbuild-launchers.
  • make check — all 21 unit suites, with new coverage for the distribution
    rule (test/unit/prted) and per-app ppr/rankfile resolution plus rewritten
    gate tests (test/unit/rmaps).
  • make -C test/offline check-offline — 1180/1180.
  • Live DVM smoke test, plus a PMIx_Spawn probe confirming PMIX_MAPPER is
    refused at both job and app level and the DVM survives.
  • contrib/dockerswarm/run-tests.sh linux — 381 passed, 0 failed, 2 skipped
    (both PMIx-capability gates), including 7 new test_rmaps cases covering
    the distribution rule and two apps placed by two components.

A placement directive written on one app of an MPMD command line was
applied to every app of the job.  Asking for one app to be mapped
differently silently mapped them all that way, and there was no way to
say what was plainly meant.

The rule is now that the first app segment is where a command line
speaks for the job: a directive written there and nowhere else describes
the whole job, however many apps follow - that is what a single-app line
has, and it is where a qualifier that spans the job can still be
written.  Written on any later app it describes that app alone.  Its
silent siblings are not agreeing with it; they said nothing, and what an
app that says nothing gets is the default.

Making that mean anything required the rest of the per-app path, which
was only half wired.  Every mapper decided whether a job was its own by
asking the *job* for the policy - but the job's policy is whatever
default was resolved for the apps that gave no directive, so seq,
rank_file and ppr all deferred and a per-app request for any of them was
quietly placed by round_robin instead.  The gates now read the resolved
options the base hands them, which in per-app dispatch is each app's own
answer.  Two apps of one job can therefore be placed by two different
components.

Three things fell out of that.  A per-app ppr pattern kept only its
count, so an app asking for two per package got two per whatever object
the job had resolved; it now carries the whole "N:object" pattern, in
the spelling the job-level attribute uses.  A per-app "--map-by
rankfile" was refused outright by the per-app parser and is now
accepted.  And the mappers that number their own procs - rank_file, seq,
lsf - each restarted from rank zero, so two apps were given the same
ranks and the second app's procs replaced the first's in jdata->procs;
they now take the base's rank cursor, and a per-app rankfile or sequence
file numbers that app's ranks into the job's numbering.

While in here, remove the pretence that a mapping component can be
chosen by name.  The mapping policy IS that choice - each component
claims the policies it implements - so a request naming a component says
nothing the policy has not already said, and when the two disagreed
there was no answer that could be right.  req_mapper and last_mapper are
gone from prte_job_map_t, and with them from the launch message: mapping
happens only on the HNP, so both strings were shipped to every daemon on
every launch and read by nothing at the far end.  PMIX_MAPPER is refused
with PMIX_ERR_NOT_SUPPORTED and is no longer advertised in the spawn
attribute lists or in PMIX_QUERY_SPAWN_SUPPORT - advertising an
attribute and then refusing it is worse than not having it.

An "rmaps" MCA parameter naming one component cannot be refused, because
which components load is settled when the framework is opened, before
any job exists.  A restriction that cannot serve the job's policy now
fails the job with a message naming what is loaded and what was asked
for, rather than the generic "none of the available mappers was able
to", which sends the reader looking at their --map-by.

Which component placed an app is still recorded, on the app itself and
only on the HNP, and --display devel-map reports it per app.

Two unrelated fixes ride along.  The dockerswarm harness asserted output
relayed back to a tool without checking PMIX_CAP_IOF_DELIVER_LOCAL; the
relay is compiled out without it, so those cases could only ever fail
against an older PMIx while the two beside them passed for the wrong
reason.  And the prohibition on mixed-version DVMs is now written down -
it is what licenses changing a packed field without a format version.

Signed-off-by: Ralph Castain <rhc@pmix.org>
@rhc54
rhc54 merged commit 828c951 into openpmix:master Aug 1, 2026
17 checks passed
@rhc54
rhc54 deleted the topic/per-app-mapping branch August 1, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rmaps: req_mapper/last_mapper are job-level, but mapping is now done per app_context

1 participant