Skip to content

hwloc: report cpu numbers in the basis we claim, reset NUMA placement counters, honor a qualifier-only --bind-to - #2597

Merged
rhc54 merged 6 commits into
openpmix:masterfrom
rhc54:topic/hwloc-review-2
Aug 2, 2026
Merged

hwloc: report cpu numbers in the basis we claim, reset NUMA placement counters, honor a qualifier-only --bind-to#2597
rhc54 merged 6 commits into
openpmix:masterfrom
rhc54:topic/hwloc-review-2

Conversation

@rhc54

@rhc54 rhc54 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

A second review pass over src/hwloc, following #2578. Three substantive
defects, three small ones, and two memory leaks the review turned up
elsewhere. Six commits, each standalone and bisectable.

The cpu numbers we print are not the ones we accept

The bits of an hwloc cpuset are always PU OS indices — the kernel's cpu
numbers. What PRRTE shows a user is a list of cores (or of hwthreads, when
the job treats hwthreads as cpus), and every PRRTE grammar that accepts
such a list — --cpu-set, the rankfile and sequence-file slot lists —
resolves it as an hwloc logical index. Those three numbering schemes
coincide only by luck.

Three renderers short-cut whenever the bits "already were" cores
(npus == ncores) or the job was in hwthreads: they printed the raw bitmap
under a core:L/hwt:L label, and ignored physical outright. So on any
SMT node viewed in hwthreads, or any node whose firmware interleaves cpu
numbers across packages, --display cpus reported numbers that --cpu-set
would resolve to a different set of cpus.

Everything now resolves each set bit to the object that owns it, through one
new primitive, prte_hwloc_base_cpuset2ranges(). prte_hwloc_build_map() is
gone with the short cut — it produced a bitmap of core indices, which a
caller then printed with hwloc_bitmap_list_snprintf(), mixing two index
bases in one string. prte_proc_t.cpuset deliberately stays raw: it is a
wire format read back with hwloc_bitmap_list_sscanf().

The same defect was in the collision messages of rmaps/seq,
rmaps/rank_file and rmaps/lsf, which printed a raw bitmap as the
"available" and "overlapping" cpus beside the user's own logical slot list.
Fixed there too.

--bind-to numa:limit=N works once per DVM

prte_hwloc_base_reset_counters() clears the per-object placement counters
between jobs. It walked only the normal depth hierarchy — and hwloc 2.x keeps
NUMA nodes out of that hierarchy, at HWLOC_TYPE_DEPTH_NUMANODE. Its
HWLOC_OBJ_NUMANODE != type guard could never fire, so a counter attached to
a NUMA node was never reset. In a persistent DVM it accumulated for the life
of the DVM, and the second --bind-to numa:limit=N job found every
domain already at its limit and could not be bound. prterun hides it by
starting a fresh HNP each time.

--bind-to :overload-allowed silently meant --bind-to none

pmix_check_cli_option() compares only min(strlen(a), strlen(b))
characters, so the empty policy word of the :qualifier form matched the
first option it was tested against — none. Binding was disabled, and
because the default mapping policy is derived from the binding, BYCORE
became BYSLOT along with it. Nothing was printed to say so.
--map-by :OVERSUBSCRIBE has always worked correctly; these now agree.

Two leaks

Found with valgrind while checking the above; both pre-existing and outside
src/hwloc, so they are the first two commits.

  • prte_relm_start_msg() unloaded the caller's pmix_data_buffer_t into
    a fresh RELM-framed one and never released the emptied container. It owns
    the buffer once it succeeds — the contract prte_rml_send_buffer_nb()
    follows and every PRTE_RML_RELIABLE_SEND caller depends on. This was one
    leaked buffer per inter-daemon reliable message, for the life of the
    DVM, not the one-off the first reproducer suggested.
  • prte_parse_locals() has three exits and only the success path
    released temp_argv/env. Any command line whose final segment fails to
    parse leaked both — --display map --display cpus is enough, since a
    repeated option is refused right there.

Smaller things

A core-less topology (PPC64 on old kernels reports only NUMA nodes and PUs)
now renders in hwthreads and says so, instead of resolving nothing and coming
back UNBOUND. A package:core id resolves inside its package rather than
by "objects-per-package × package id". prte_hwloc_print() and its three
callers handle a node whose topology has not arrived. The
hwloc_default_cpu_list MCA value is no longer truncated in place. Plus a
NULL PU deref and an info-array removal loop that skipped the entry shifted
into the vacated slot.

Testing

  • make checktest/unit/hwloc extended; the logical-vs-physical case
    needs a hand-written XML topology, because hwloc's synthetic generator
    always makes os_index and logical_index agree. (Note for anyone
    extending it: hwloc's XML importer segfaults rather than diagnoses on an
    object missing complete_cpuset/nodeset/complete_nodeset.)
  • make -C test/offline check-offline — 1180/1180.
  • contrib/dockerswarmtest_hwloc grew from 23 to 35 cases, including
    the one that needs a persistent DVM (three successive numa:limit=1
    jobs) and the print-then-accept round trip. Full suite: 407 passed, 0
    failed, 2 skipped
    (both skips pre-existing PMIx-capability gates).
  • valgrind on the rendering and launch paths: definitely lost: 0.

Each commit builds warning-free on its own.

🤖 Generated with Claude Code

rhc54 added 6 commits August 1, 2026 19:40
prte_relm_start_msg() frames the caller's message into a new buffer by
unloading the payload out of the one it was handed and packing it, with
the RML tag, into a fresh buffer that the caddy then owns.  Unloading a
buffer moves its payload out but leaves the container behind, and nothing
released it.

The function owns the caller's buffer once it succeeds - the same contract
prte_rml_send_buffer_nb() follows, and the one every PRTE_RML_RELIABLE_SEND
call site depends on, since none of them releases the buffer after a
successful send.  So a pmix_data_buffer_t was leaked on every reliable
message sent between daemons, for the life of the DVM.

Valgrind names this shape clearly if you know to look: a record with direct
bytes but zero indirect bytes means a container outlived a payload that had
been moved out of it.

Signed-off-by: Ralph Castain <rhc@pmix.org>
prte_parse_locals() has three exits: a create_app() failure inside the
segment loop, a create_app() failure for the trailing segment, and the
success path.  Only the last released temp_argv and env, so any command
line whose final segment fails to parse leaked both.

Reaching that is not obscure - a repeated option is refused right there,
so "prterun -n 1 --display map --display cpus hostname" is enough.  The
in-loop exit was releasing temp_argv but not env, which create_app() may
already have filled in before it failed.

Record in contrib/dockerswarm/AGENTS.md that a leak hunt has to run a
command line that fails, not only one that works, and that "definitely
lost" is expected to be zero.

Signed-off-by: Ralph Castain <rhc@pmix.org>
Binding attaches a prte_hwloc_obj_data_t to each candidate object to
record how many procs it has already placed there, and
prte_hwloc_base_reset_counters() clears those between jobs.  It walked
only the normal depth hierarchy, 1..hwloc_topology_get_depth() - but
hwloc 2.x keeps NUMA nodes out of that hierarchy entirely, at the special
depth HWLOC_TYPE_DEPTH_NUMANODE.  The HWLOC_OBJ_NUMANODE test in its
type filter could therefore never fire, and a counter attached to a NUMA
node was never cleared.

"--bind-to numa:limit=N" is what attaches one.  In a persistent DVM the
counter accumulated for the life of the DVM, so the second such job found
every domain already at its limit, skipped them all, and could not be
bound.  prterun hides it by starting a fresh HNP each time.

Sweep that depth separately, exactly as prte_hwloc_base_release_userdata()
already has to, and drop the unreachable test.

Signed-off-by: Ralph Castain <rhc@pmix.org>
The bits of an hwloc cpuset are always PU OS indices - the kernel's cpu
numbers.  What PRRTE reports to a user is a list of cores, or of hwthreads
when the job treats hwthreads as cpus, and every PRRTE grammar that
accepts such a list - --cpu-set, the rankfile and sequence-file slot
lists - resolves it as an hwloc logical index.  Those three numbering
schemes coincide only by luck.

Three renderers took a short cut whenever the bits "already were" cores
(npus == ncores) or the job was in hwthreads: they printed the raw bitmap
under a "core:L"/"hwt:L" label and ignored the caller's "physical" flag
outright.  That is wrong on precisely the machines where the answer
matters - any SMT node viewed in hwthreads, and any node whose firmware
interleaves cpu numbers across packages - and it is worse than cosmetic,
because the user reads the number back out and hands it to --cpu-set.

Resolve each set bit to the object that owns it and take that object's
index.  prte_hwloc_base_cpuset2ranges() is the one place that does it;
prte_hwloc_base_cset2str(), prte_hwloc_get_binding_info() (which gains a
"physical" parameter its caller was already computing and discarding), and
both copies of --display cpus now go through it, as do the "available" and
"overlapping" cpu sets that rmaps/seq, rmaps/rank_file and rmaps/lsf print
beside the user's own logical slot list when it collides.

prte_hwloc_build_map() is gone with the short cut: it produced a bitmap of
core indices, which a caller then printed with hwloc_bitmap_list_snprintf(),
mixing two index bases in one string.  prte_proc_t.cpuset stays raw - it is
a wire format read back with hwloc_bitmap_list_sscanf().

A topology with PUs but no cores is now rendered in hwthreads and says so,
rather than resolving nothing and coming back UNBOUND: hwloc does not find
cores on every platform, and prte_hwloc_base_get_pu() has always allowed
for it.

The unit test drives the divergence from a hand-written XML topology whose
PU OS indices interleave across two packages - hwloc's synthetic generator
always makes os_index and logical_index agree, so XML is the only way to
prove a renderer reports the basis it claims.

Signed-off-by: Ralph Castain <rhc@pmix.org>
"--bind-to :overload-allowed" means "whatever binding I would otherwise
have got, but allow overload", exactly as "--map-by :OVERSUBSCRIBE" has
always meant on the mapping side.  It did not.

pmix_check_cli_option() compares only min(strlen(a), strlen(b))
characters, so an empty string matches whatever it is tested against
first - and for the ":qualifier" form the policy word is empty, so it fell
into the first arm of the chain, "none".  Binding was silently disabled,
and because the default mapping policy is derived from the binding it took
that down with it too: BYCORE became BYSLOT.  Nothing was printed to say
any of this had happened.

Resolve the policy word before applying any qualifier, and leave the policy
bits at zero when there is no policy word, so PRTE_BINDING_POLICY_IS_SET()
still answers "no" and PRTE_SET_DEFAULT_BINDING_POLICY() fills the policy
in around the qualifiers later.

Doing the policy first also fixes a smaller thing: "report" and "limit=N"
write to jdata->attributes, so parsing them ahead of the policy meant
"--bind-to sockets:report" recorded report-bindings on the job and then
rejected the request.

Signed-off-by: Ralph Castain <rhc@pmix.org>
Six small fixes in src/hwloc, none of which needs its own commit:

prte_hwloc_base_core_cpus() dereferenced the topology's PU 0 without
checking it, having just checked core 0.

prte_hwloc_base_get_topology()'s removal of the HostName/ProcessName info
entries left-justified the array and then advanced its index, so the entry
that had just shifted into the vacated slot was never examined - two
removable entries side by side left the second one behind.  It also
matched by prefix, so "HostNameFoo" counted.

prte_hwloc_base_register() split the ":HWTCPUS" modifier off the
hwloc_default_cpu_list value in place.  The MCA layer owns that string and
reports the parameter's value through it, so prte_info and every later
reader saw a truncated value the user never set.  Parse a copy.

package_core_to_cpu_set() resolved a package-relative core id by adding
"objects per package times package id" to a global index, which names an
object in the wrong package as soon as two packages hold different numbers
of them.  hwloc_get_obj_inside_cpuset_by_type() asks the question directly.

prte_hwloc_print() had no guard for a NULL topology or a topology with no
root, and its three callers printed *output unconditionally - a node in the
pool holds no topology until its daemon reports in, and both --display topo
and the ess/hnp verbose dump are pool walks.

prte_hwloc_base_generate_cpuset() recomputed PMIx_Argv_count() on every
iteration of a loop that cannot change the array.

Signed-off-by: Ralph Castain <rhc@pmix.org>
@rhc54
rhc54 merged commit 3441a8c into openpmix:master Aug 2, 2026
17 checks passed
@rhc54
rhc54 deleted the topic/hwloc-review-2 branch August 2, 2026 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant