Add Cray/HPE pm_counters backend with cross-vendor GPU correlation hints by vancraar · Pull Request #9 · SC-SGS/hardware_sampling

vancraar · 2026-08-13T14:23:56Z

Cray/HPE pm_counters backend

New cray_pm_counters backend reading whole-node and per-accelerator
power/energy from Cray/HPE's /sys/cray/pm_counters sysfs interface, to
validate hws's software-side energy measurements against measured
hardware ground truth on HLRS's Hunter system (HPE Cray EX, AMD MI300A
APU nodes). The counter set is discovered and classified at runtime
instead of hardcoded, since the exposed files differ across PM counter
versions and node generations. Confirmed against a real Hunter mi300a
node (23 files, PM counters v3). Includes CTest-based regression tests
using fixtures from that real dump.

Cross-vendor accel[i] <-> GPU correlation hints

pm_counters exposes per-accelerator counters as accel[i]_energy/
accel[i]_power, but doesn't say which physical GPU (as seen by
HIP/CUDA) each index corresponds to. Adds a best-effort correlation
guess, generalized from AMD-only to also cover NVIDIA (pm_counters uses
the same accel[i] naming convention on Grace-Hopper/EX254n nodes):
gpu_amd_hardware_sampler/gpu_nvidia_hardware_sampler each gained a
pci_bus_id() querying the same handle already used for their actual
measurements, and a shared, vendor-agnostic
accel_correlation_yaml_block() builds the resulting YAML hint block
that system_hardware_sampler appends to dump_yaml/dump_yaml_global.

Python bindings for cray_pm_counters

Mirrors the existing per-backend binding pattern (one .cpp file,
HWS_FOR_CRAY_PM_COUNTERS_ENABLED-gated). Built and tested locally and
on a real Hunter mi300a node via a PBS job, including a dump_yaml
roundtrip with the correlation-hints section.

Known limitations

The NVIDIA-side correlation code (pci_bus_id(),
enumerate_all_nvidia_gpu_pci_bus_ids()) currently sits on top of the
same NVML device-index-resolution bug fixed in Fix GPU device index resolution and init/shutdown lifecycle races #8
(fix/gpu-device-index-resolution) - this branch still has its own,
earlier, entangled copy of the AMD half of that fix (commit a029bd0)
predating the standalone, more thoroughly reviewed fix in Fix GPU device index resolution and init/shutdown lifecycle races #8. Once Fix GPU device index resolution and init/shutdown lifecycle races #8
merges, this branch needs a rebase onto the new develop to drop the
duplicate and pick up the reviewed NVIDIA/lifecycle fixes cleanly.
MI300A partitioned accelerators and NVIDIA MIG instances aren't
disambiguated by the correlation guess (same limitation as Fix GPU device index resolution and init/shutdown lifecycle races #8).

Also included

A pre-existing, unrelated fix already merged onto this branch: turbostat
subprocess deadlock and unreliable short sampling intervals in the CPU
backend (commit 1884d03).

Reads whole-node power/energy from Cray/HPE's /sys/cray/pm_counters sysfs interface, to validate hws's software energy measurements against measured hardware ground truth on HLRS's Hunter system. The counter set is discovered and classified at runtime instead of hardcoded, since the exposed files differ across PM counter versions and node generations; confirmed against a real Hunter mi300a node (23 files, PM counters v3). Includes CTest-based regression tests using fixtures from that real dump.

Generalizes the pm_counters accel[i] index guess from AMD-only to also cover NVIDIA, since pm_counters exposes the same accel[i] naming convention on Grace-Hopper (EX254n) nodes. Extracts the vendor-specific YAML formatting into a shared, vendor-agnostic hws::detail::accel_correlation_yaml_block() so AMD and NVIDIA share one code path in system_hardware_sampler's device_correlation_hints. Both gpu_amd_hardware_sampler and gpu_nvidia_hardware_sampler gained a pci_bus_id() that queries the same API/handle each sampler already uses for its actual measurements (ROCm SMI's rsmi_dev_pci_id_get(device_id_), NVML's nvmlDeviceGetPciInfo_v3() on the already-resolved handle) instead of a different API family (HIP/CUDA's own PCI bus ID calls). Those differ from ROCm SMI's/NVML's device enumeration under HIP_VISIBLE_DEVICES/CUDA_VISIBLE_DEVICES, which could otherwise silently attribute a correlation guess to the wrong physical device.

dump_yaml()/dump_yaml_global() wrote each sampler's own YAML document directly instead of going through as_yaml_string(), so device_correlation_hints_as_yaml_string() never made it into a dumped YAML file even though as_yaml_string() included it correctly. Found by running the actual example program on a real Hunter mi300a node - the correlation section was silently missing from track.yaml. dump_yaml(std::string)/dump_yaml(path) now forward to dump_yaml(const char*) instead of duplicating the sampler loop.

…vals The CPU turbostat backend could deadlock: run_subprocess() never combined the child's stdout/stderr, so an unread stderr pipe could fill up and block the child forever, wedging subprocess_join() (reproduced via gdb/wchan on two architectures). Fixed by passing subprocess_option_combined_stdout_stderr. That fix has a side effect: turbostat's own diagnostic messages (e.g. "Disabling Low Power Idle CPU output") now land in the same stream hws parses, which used to assume the header/value pair were always the first two lines. Parsing now takes the *last* two lines instead, since turbostat always prints diagnostics before its one-shot table, never after. Also fixed: turbostat's -i interval was hardcoded to 0.001s (1ms), which empirically (four real machines, full-core load) causes physically implausible readings (multi-kW power, multi-GHz clocks) because turbostat's own internal timing calculation degrades under load at short intervals. This is now the configurable HWS_TURBOSTAT_INTERVAL CMake option; the default stays at 0.001 for backwards compatibility with existing sudoers allowlists and paper-reproducibility setups, but a longer interval (e.g. 1s) is documented as the reliable choice. Related changes: - warn at runtime if sampling_interval is shorter than HWS_TURBOSTAT_INTERVAL - sleep_until() instead of sleep_for() so a slow turbostat call doesn't also get a full extra sampling_interval tacked on afterward - only invoke turbostat if an enabled sample_category actually needs it

gpu_amd_hardware_sampler used the HIP-relative device index directly as the ROCm SMI index for every rsmi_dev_* call. ROCm SMI's own device enumeration isn't filtered by HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES the way HIP's is, so under a non-default visibility mask the sampler measured the wrong physical GPU. Fix: resolve the actual ROCm SMI index once at construction via a PCI-bus-ID match against every ROCm SMI device, throwing if no match is found instead of silently falling back to the HIP-relative index. The original HIP-relative index is kept in a new hip_device_id_ member for the one HIP call that still needs HIP-space, and for system_hardware_sampler's device_correlation_hints output. Also adds the missing find_dependency(MPI) to hwsConfig.cmake.in, alongside the existing ones for CUDA/HIP/ROCm-SMI/Level-Zero.

Mirrors the existing per-backend binding pattern: one .cpp file (bindings/cray_pm_counters_hardware_sampler.cpp) binding CrayPmCountersGeneralSamples, CrayPmCountersPowerSamples and CrayPmCountersHardwareSampler (constructors, general_samples, power_samples, discovered_accel_indices, samples_only_as_yaml_string, __repr__), gated behind HWS_FOR_CRAY_PM_COUNTERS_ENABLED like every other backend's bindings. device_identification() stays unbound, consistent with every other backend. Built and tested locally (non-MPI build) and on a real Hunter mi300a node via a PBS job: SystemHardwareSampler auto-discovery, direct CrayPmCountersHardwareSampler construction, and a dump_yaml roundtrip including the device_correlation_hints section all work correctly.

breyerml · 2026-08-23T12:23:48Z

    void create_local_samplers(std::chrono::milliseconds sampling_interval, hws::sample_category category);
+
+    /**
+     * @brief Generate a best-effort, UNVERIFIED YAML hint correlating each visible AMD GPU device with a Cray


UNVERIFIED?

breyerml · 2026-08-23T12:24:34Z

Should this HLRS-specific file be checked in?

breyerml · 2026-08-23T12:26:44Z

+    // turbostat itself blocks for HWS_TURBOSTAT_INTERVAL seconds per invocation -> if that's
+    // longer than the requested sampling interval, the turbostat backend will dominate and the
+    // achieved cadence will be closer to HWS_TURBOSTAT_INTERVAL than to sampling_interval
+    const double turbostat_interval_seconds = std::stod(HWS_TURBOSTAT_INTERVAL);


If the HWS_TURBOSTAT_INTERVAL is not passed to C++ as string, this conversion would be unnecessary.

breyerml · 2026-08-23T12:29:20Z

+ *          `hws::detail::format_pci_bus_id()`.
+ */
+[[nodiscard]] std::string bdfid_to_pci_bus_id(const std::uint64_t bdfid) {
+    const auto domain = static_cast<std::uint32_t>((bdfid >> 32) & 0xffffffffull);


This PR contains the same changes as the other currently open PR? -> see my comments there.

breyerml · 2026-08-23T12:31:18Z

Would be very nice to instead use a proper testing framework and also include tests for the other hardware samplers (possibly in another PR).

breyerml · 2026-08-23T12:31:57Z

    return std::accumulate(samplers_.cbegin(), samplers_.cend(), std::string{}, [](const std::string str, const auto &ptr) { return str + ptr->samples_only_as_yaml_string(); });
 }

+#if defined(HWS_FOR_CRAY_PM_COUNTERS_ENABLED) && (defined(HWS_FOR_AMD_GPUS_ENABLED) || defined(HWS_FOR_NVIDIA_GPUS_ENABLED))


Not supported for Intel GPUs?

breyerml · 2026-08-23T12:33:24Z

+    #endif
+    }
+
+    const bool any_visible_gpus =


I really dislike this block. Why not create the vectors even if the defines are not there and just check for emptiness here?

breyerml · 2026-08-23T12:37:20Z

@@ -227,7 +239,7 @@ current clock frequencies, temperatures, or memory consumption.
 | sample                  | sample type | CPUs | NVIDIA GPUs | AMD GPUs | Intel GPUs |


I guess this table should also be updated.

breyerml · 2026-08-23T12:38:21Z

Please document new dependency.

vancraar added 7 commits August 11, 2026 11:09

Merge turbostat subprocess deadlock and interval jitter fix

6c36a23

vancraar requested a review from TimThuering August 13, 2026 14:26

breyerml reviewed Aug 23, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add Cray/HPE pm_counters backend with cross-vendor GPU correlation hints - #9

Add Cray/HPE pm_counters backend with cross-vendor GPU correlation hints#9
vancraar wants to merge 7 commits into
developfrom
feature/cray-pm-counters

vancraar commented Aug 13, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

breyerml Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -227,7 +239,7 @@ current clock frequencies, temperatures, or memory consumption.
		\| sample \| sample type \| CPUs \| NVIDIA GPUs \| AMD GPUs \| Intel GPUs \|

Uh oh!

Conversation

vancraar commented Aug 13, 2026

Cray/HPE pm_counters backend

Cross-vendor accel[i] <-> GPU correlation hints

Python bindings for cray_pm_counters

Known limitations

Also included

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants