hwloc: report cpu numbers in the basis we claim, reset NUMA placement counters, honor a qualifier-only --bind-to - #2597
Merged
Conversation
prte_relm_start_msg() frames the caller's message into a new buffer by unloading the payload out of the one it was handed and packing it, with the RML tag, into a fresh buffer that the caddy then owns. Unloading a buffer moves its payload out but leaves the container behind, and nothing released it. The function owns the caller's buffer once it succeeds - the same contract prte_rml_send_buffer_nb() follows, and the one every PRTE_RML_RELIABLE_SEND call site depends on, since none of them releases the buffer after a successful send. So a pmix_data_buffer_t was leaked on every reliable message sent between daemons, for the life of the DVM. Valgrind names this shape clearly if you know to look: a record with direct bytes but zero indirect bytes means a container outlived a payload that had been moved out of it. Signed-off-by: Ralph Castain <rhc@pmix.org>
prte_parse_locals() has three exits: a create_app() failure inside the segment loop, a create_app() failure for the trailing segment, and the success path. Only the last released temp_argv and env, so any command line whose final segment fails to parse leaked both. Reaching that is not obscure - a repeated option is refused right there, so "prterun -n 1 --display map --display cpus hostname" is enough. The in-loop exit was releasing temp_argv but not env, which create_app() may already have filled in before it failed. Record in contrib/dockerswarm/AGENTS.md that a leak hunt has to run a command line that fails, not only one that works, and that "definitely lost" is expected to be zero. Signed-off-by: Ralph Castain <rhc@pmix.org>
Binding attaches a prte_hwloc_obj_data_t to each candidate object to record how many procs it has already placed there, and prte_hwloc_base_reset_counters() clears those between jobs. It walked only the normal depth hierarchy, 1..hwloc_topology_get_depth() - but hwloc 2.x keeps NUMA nodes out of that hierarchy entirely, at the special depth HWLOC_TYPE_DEPTH_NUMANODE. The HWLOC_OBJ_NUMANODE test in its type filter could therefore never fire, and a counter attached to a NUMA node was never cleared. "--bind-to numa:limit=N" is what attaches one. In a persistent DVM the counter accumulated for the life of the DVM, so the second such job found every domain already at its limit, skipped them all, and could not be bound. prterun hides it by starting a fresh HNP each time. Sweep that depth separately, exactly as prte_hwloc_base_release_userdata() already has to, and drop the unreachable test. Signed-off-by: Ralph Castain <rhc@pmix.org>
The bits of an hwloc cpuset are always PU OS indices - the kernel's cpu numbers. What PRRTE reports to a user is a list of cores, or of hwthreads when the job treats hwthreads as cpus, and every PRRTE grammar that accepts such a list - --cpu-set, the rankfile and sequence-file slot lists - resolves it as an hwloc logical index. Those three numbering schemes coincide only by luck. Three renderers took a short cut whenever the bits "already were" cores (npus == ncores) or the job was in hwthreads: they printed the raw bitmap under a "core:L"/"hwt:L" label and ignored the caller's "physical" flag outright. That is wrong on precisely the machines where the answer matters - any SMT node viewed in hwthreads, and any node whose firmware interleaves cpu numbers across packages - and it is worse than cosmetic, because the user reads the number back out and hands it to --cpu-set. Resolve each set bit to the object that owns it and take that object's index. prte_hwloc_base_cpuset2ranges() is the one place that does it; prte_hwloc_base_cset2str(), prte_hwloc_get_binding_info() (which gains a "physical" parameter its caller was already computing and discarding), and both copies of --display cpus now go through it, as do the "available" and "overlapping" cpu sets that rmaps/seq, rmaps/rank_file and rmaps/lsf print beside the user's own logical slot list when it collides. prte_hwloc_build_map() is gone with the short cut: it produced a bitmap of core indices, which a caller then printed with hwloc_bitmap_list_snprintf(), mixing two index bases in one string. prte_proc_t.cpuset stays raw - it is a wire format read back with hwloc_bitmap_list_sscanf(). A topology with PUs but no cores is now rendered in hwthreads and says so, rather than resolving nothing and coming back UNBOUND: hwloc does not find cores on every platform, and prte_hwloc_base_get_pu() has always allowed for it. The unit test drives the divergence from a hand-written XML topology whose PU OS indices interleave across two packages - hwloc's synthetic generator always makes os_index and logical_index agree, so XML is the only way to prove a renderer reports the basis it claims. Signed-off-by: Ralph Castain <rhc@pmix.org>
"--bind-to :overload-allowed" means "whatever binding I would otherwise have got, but allow overload", exactly as "--map-by :OVERSUBSCRIBE" has always meant on the mapping side. It did not. pmix_check_cli_option() compares only min(strlen(a), strlen(b)) characters, so an empty string matches whatever it is tested against first - and for the ":qualifier" form the policy word is empty, so it fell into the first arm of the chain, "none". Binding was silently disabled, and because the default mapping policy is derived from the binding it took that down with it too: BYCORE became BYSLOT. Nothing was printed to say any of this had happened. Resolve the policy word before applying any qualifier, and leave the policy bits at zero when there is no policy word, so PRTE_BINDING_POLICY_IS_SET() still answers "no" and PRTE_SET_DEFAULT_BINDING_POLICY() fills the policy in around the qualifiers later. Doing the policy first also fixes a smaller thing: "report" and "limit=N" write to jdata->attributes, so parsing them ahead of the policy meant "--bind-to sockets:report" recorded report-bindings on the job and then rejected the request. Signed-off-by: Ralph Castain <rhc@pmix.org>
Six small fixes in src/hwloc, none of which needs its own commit: prte_hwloc_base_core_cpus() dereferenced the topology's PU 0 without checking it, having just checked core 0. prte_hwloc_base_get_topology()'s removal of the HostName/ProcessName info entries left-justified the array and then advanced its index, so the entry that had just shifted into the vacated slot was never examined - two removable entries side by side left the second one behind. It also matched by prefix, so "HostNameFoo" counted. prte_hwloc_base_register() split the ":HWTCPUS" modifier off the hwloc_default_cpu_list value in place. The MCA layer owns that string and reports the parameter's value through it, so prte_info and every later reader saw a truncated value the user never set. Parse a copy. package_core_to_cpu_set() resolved a package-relative core id by adding "objects per package times package id" to a global index, which names an object in the wrong package as soon as two packages hold different numbers of them. hwloc_get_obj_inside_cpuset_by_type() asks the question directly. prte_hwloc_print() had no guard for a NULL topology or a topology with no root, and its three callers printed *output unconditionally - a node in the pool holds no topology until its daemon reports in, and both --display topo and the ess/hnp verbose dump are pool walks. prte_hwloc_base_generate_cpuset() recomputed PMIx_Argv_count() on every iteration of a loop that cannot change the array. Signed-off-by: Ralph Castain <rhc@pmix.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A second review pass over
src/hwloc, following #2578. Three substantivedefects, three small ones, and two memory leaks the review turned up
elsewhere. Six commits, each standalone and bisectable.
The cpu numbers we print are not the ones we accept
The bits of an hwloc cpuset are always PU OS indices — the kernel's cpu
numbers. What PRRTE shows a user is a list of cores (or of hwthreads, when
the job treats hwthreads as cpus), and every PRRTE grammar that accepts
such a list —
--cpu-set, the rankfile and sequence-file slot lists —resolves it as an hwloc logical index. Those three numbering schemes
coincide only by luck.
Three renderers short-cut whenever the bits "already were" cores
(
npus == ncores) or the job was in hwthreads: they printed the raw bitmapunder a
core:L/hwt:Llabel, and ignoredphysicaloutright. So on anySMT node viewed in hwthreads, or any node whose firmware interleaves cpu
numbers across packages,
--display cpusreported numbers that--cpu-setwould resolve to a different set of cpus.
Everything now resolves each set bit to the object that owns it, through one
new primitive,
prte_hwloc_base_cpuset2ranges().prte_hwloc_build_map()isgone with the short cut — it produced a bitmap of core indices, which a
caller then printed with
hwloc_bitmap_list_snprintf(), mixing two indexbases in one string.
prte_proc_t.cpusetdeliberately stays raw: it is awire format read back with
hwloc_bitmap_list_sscanf().The same defect was in the collision messages of
rmaps/seq,rmaps/rank_fileandrmaps/lsf, which printed a raw bitmap as the"available" and "overlapping" cpus beside the user's own logical slot list.
Fixed there too.
--bind-to numa:limit=Nworks once per DVMprte_hwloc_base_reset_counters()clears the per-object placement countersbetween jobs. It walked only the normal depth hierarchy — and hwloc 2.x keeps
NUMA nodes out of that hierarchy, at
HWLOC_TYPE_DEPTH_NUMANODE. ItsHWLOC_OBJ_NUMANODE != typeguard could never fire, so a counter attached toa NUMA node was never reset. In a persistent DVM it accumulated for the life
of the DVM, and the second
--bind-to numa:limit=Njob found everydomain already at its limit and could not be bound.
prterunhides it bystarting a fresh HNP each time.
--bind-to :overload-allowedsilently meant--bind-to nonepmix_check_cli_option()compares onlymin(strlen(a), strlen(b))characters, so the empty policy word of the
:qualifierform matched thefirst option it was tested against —
none. Binding was disabled, andbecause the default mapping policy is derived from the binding,
BYCOREbecame
BYSLOTalong with it. Nothing was printed to say so.--map-by :OVERSUBSCRIBEhas always worked correctly; these now agree.Two leaks
Found with valgrind while checking the above; both pre-existing and outside
src/hwloc, so they are the first two commits.prte_relm_start_msg()unloaded the caller'spmix_data_buffer_tintoa fresh RELM-framed one and never released the emptied container. It owns
the buffer once it succeeds — the contract
prte_rml_send_buffer_nb()follows and every
PRTE_RML_RELIABLE_SENDcaller depends on. This was oneleaked buffer per inter-daemon reliable message, for the life of the
DVM, not the one-off the first reproducer suggested.
prte_parse_locals()has three exits and only the success pathreleased
temp_argv/env. Any command line whose final segment fails toparse leaked both —
--display map --display cpusis enough, since arepeated option is refused right there.
Smaller things
A core-less topology (PPC64 on old kernels reports only NUMA nodes and PUs)
now renders in hwthreads and says so, instead of resolving nothing and coming
back
UNBOUND. Apackage:coreid resolves inside its package rather thanby "objects-per-package × package id".
prte_hwloc_print()and its threecallers handle a node whose topology has not arrived. The
hwloc_default_cpu_listMCA value is no longer truncated in place. Plus aNULL PU deref and an info-array removal loop that skipped the entry shifted
into the vacated slot.
Testing
make check—test/unit/hwlocextended; the logical-vs-physical caseneeds a hand-written XML topology, because hwloc's synthetic generator
always makes
os_indexandlogical_indexagree. (Note for anyoneextending it: hwloc's XML importer segfaults rather than diagnoses on an
object missing
complete_cpuset/nodeset/complete_nodeset.)make -C test/offline check-offline— 1180/1180.contrib/dockerswarm—test_hwlocgrew from 23 to 35 cases, includingthe one that needs a persistent DVM (three successive
numa:limit=1jobs) and the print-then-accept round trip. Full suite: 407 passed, 0
failed, 2 skipped (both skips pre-existing PMIx-capability gates).
definitely lost: 0.Each commit builds warning-free on its own.
🤖 Generated with Claude Code