Place each app of a job by the directive it was given - #2594
Merged
Conversation
A placement directive written on one app of an MPMD command line was applied to every app of the job. Asking for one app to be mapped differently silently mapped them all that way, and there was no way to say what was plainly meant. The rule is now that the first app segment is where a command line speaks for the job: a directive written there and nowhere else describes the whole job, however many apps follow - that is what a single-app line has, and it is where a qualifier that spans the job can still be written. Written on any later app it describes that app alone. Its silent siblings are not agreeing with it; they said nothing, and what an app that says nothing gets is the default. Making that mean anything required the rest of the per-app path, which was only half wired. Every mapper decided whether a job was its own by asking the *job* for the policy - but the job's policy is whatever default was resolved for the apps that gave no directive, so seq, rank_file and ppr all deferred and a per-app request for any of them was quietly placed by round_robin instead. The gates now read the resolved options the base hands them, which in per-app dispatch is each app's own answer. Two apps of one job can therefore be placed by two different components. Three things fell out of that. A per-app ppr pattern kept only its count, so an app asking for two per package got two per whatever object the job had resolved; it now carries the whole "N:object" pattern, in the spelling the job-level attribute uses. A per-app "--map-by rankfile" was refused outright by the per-app parser and is now accepted. And the mappers that number their own procs - rank_file, seq, lsf - each restarted from rank zero, so two apps were given the same ranks and the second app's procs replaced the first's in jdata->procs; they now take the base's rank cursor, and a per-app rankfile or sequence file numbers that app's ranks into the job's numbering. While in here, remove the pretence that a mapping component can be chosen by name. The mapping policy IS that choice - each component claims the policies it implements - so a request naming a component says nothing the policy has not already said, and when the two disagreed there was no answer that could be right. req_mapper and last_mapper are gone from prte_job_map_t, and with them from the launch message: mapping happens only on the HNP, so both strings were shipped to every daemon on every launch and read by nothing at the far end. PMIX_MAPPER is refused with PMIX_ERR_NOT_SUPPORTED and is no longer advertised in the spawn attribute lists or in PMIX_QUERY_SPAWN_SUPPORT - advertising an attribute and then refusing it is worse than not having it. An "rmaps" MCA parameter naming one component cannot be refused, because which components load is settled when the framework is opened, before any job exists. A restriction that cannot serve the job's policy now fails the job with a message naming what is loaded and what was asked for, rather than the generic "none of the available mappers was able to", which sends the reader looking at their --map-by. Which component placed an app is still recorded, on the app itself and only on the HNP, and --display devel-map reports it per app. Two unrelated fixes ride along. The dockerswarm harness asserted output relayed back to a tool without checking PMIX_CAP_IOF_DELIVER_LOCAL; the relay is compiled out without it, so those cases could only ever fail against an older PMIx while the two beside them passed for the wrong reason. And the prohibition on mixed-version DVMs is now written down - it is what licenses changing a packed field without a format version. Signed-off-by: Ralph Castain <rhc@pmix.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2563.
The problem
A placement directive written on one app of an MPMD command line was applied
to every app of the job. Asking for one app to be mapped differently
silently mapped them all that way, and there was no way to say what was
plainly meant.
The rule
The first app segment is where a command line speaks for the job. A
directive written there and nowhere else describes the whole job, however
many apps follow — that is what a single-app line has, and it is where a
qualifier spanning the job can still be written. Written on any later app it
describes that app alone; its silent siblings take the defaults, because an
app that says nothing is not agreeing with one that did.
What #2563 turned out to require
The issue observed that
req_mapper/last_mapperare job-level whilemapping is per-app, and predicted it "will break first when per-app mapper
selection is made to work properly." It does, and the reason is that every
mapper decided whether a job was its own by asking the job for the
policy. The job's policy is whatever default was resolved for the apps that
gave no directive, so
seq,rank_fileandpprall deferred and aper-app request for any of them was quietly placed by
round_robininstead.Gates now read the resolved options the base hands them, which in per-app
dispatch is each app's own answer. Two apps of one job can therefore be
placed by two different components.
Three things fell out of that:
pprpattern kept only its count, so an app asking for twoper package got two per whatever object the job had resolved. It now
carries the whole
N:objectpattern.--map-by rankfilewas refused outright by the per-app parser.It is now accepted.
rank_file,seq,lsf— eachrestarted from rank zero, so two apps were given the same ranks and the
second app's procs replaced the first's in
jdata->procs. They now takethe base's rank cursor.
Mapper selection is the policy
req_mapperandlast_mapperare removed fromprte_job_map_t, and withthem from the launch message — mapping happens only on the HNP, so both
strings were shipped to every daemon on every launch and read by nothing at
the far end.
PMIX_MAPPERis refused withPMIX_ERR_NOT_SUPPORTEDand is no longeradvertised in the spawn attribute lists or in
PMIX_QUERY_SPAWN_SUPPORT.Naming a component says nothing
PMIX_MAPBYhas not already said, and thetwo can contradict each other with no answer that could be right.
An
rmapsMCA parameter naming one component cannot be refused — whichcomponents load is settled when the framework is opened, before any job
exists. A restriction that cannot serve the job's policy now fails the job
with a message naming what is loaded and what was asked for:
Which component placed an app is still recorded — on the app, only on the
HNP — and
--display devel-mapreports it per app.Note on the wire format
prte_map_pack/prte_map_unpackchange together in this commit. That isonly safe because mixed-version DVMs are forbidden, which was nowhere
written down; it is now, in
docs/versions.rstand the developer guides.Two unrelated fixes ride along
checking
PMIX_CAP_IOF_DELIVER_LOCAL. The relay is compiled out withoutit, so those cases could only ever fail against an older PMIx — while the
two beside them passed for the wrong reason.
Testing
--enable-debugbuild;lsfcompiled via--enable-testbuild-launchers.make check— all 21 unit suites, with new coverage for the distributionrule (
test/unit/prted) and per-app ppr/rankfile resolution plus rewrittengate tests (
test/unit/rmaps).make -C test/offline check-offline— 1180/1180.PMIx_Spawnprobe confirmingPMIX_MAPPERisrefused at both job and app level and the DVM survives.
contrib/dockerswarm/run-tests.sh linux— 381 passed, 0 failed, 2 skipped(both PMIx-capability gates), including 7 new
test_rmapscases coveringthe distribution rule and two apps placed by two components.