Affinity mapping

[Maff Glover (mo-mglover)]

Description

Rather than take on-trust that MPI ranks and threads are bound to cores as expected, perhaps according to launcher flags or OMP_ environment variables, it is useful to be able to test this directly. More specifically, test the affinity from a running application on the platform / in the environment that an application might be running.

Including this functionality in Vernier is useful because any client application will then have access to it. The new functionality is exposed via two new API calls (one C++, one Fortran). As with all Vernier calls, these can be built into an application, perhaps behind a switch.

Example:

• 7 ranks
  • One compute rank with 122 threads.
  • 6 I/O server ranks. (Single-threaded.)
• Each thread bound to one physical core only.
  • Virtual cores are empty.
  • This does need some knowledge of how cores are numbered on a node. lscpu will give that information.

--> AFFINITY MAP <--

Maximum number of (logical) cores: 256

Thread binding map, key:

    . = No threads running on this core.
    # = Multiple threads running on this core.
  0-9 = Threads 0-9.
  a-z = Threads 10-35.
  A-Z = Threads 36-61.
    $ = Threads 62 and greater.

MPI rank : ...THREADS..ON..CORES... : Num. cores available for migration.

           Cores ---->
00000000 : 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$...................................................................................................................................... :   1
00000001 : ..........................................................................................................................0..................................................................................................................................... :   1
00000002 : ...........................................................................................................................0.................................................................................................................................... :   1
00000003 : ............................................................................................................................0................................................................................................................................... :   1
00000004 : .............................................................................................................................0.................................................................................................................................. :   1
00000005 : ..............................................................................................................................0................................................................................................................................. :   1
00000006 : ...............................................................................................................................0................................................................................................................................ :   1

Linked issues

No associated issue.

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• New tests have been added
• Tests have been modified to accommodate this change

Checklist:

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes, for both debug and optimised builds

[Copilot]

Pull request overview

Adds an “affinity mapping” feature to Vernier so client applications (C++ and Fortran) can write an ASCII-art report showing where MPI ranks and threads are running/bound at runtime.

Changes:

• Introduces a new C++ affinity subsystem (Affinity, AffinitySysCalls) and wires it into the Vernier public API (write_affinity*).
• Extends Fortran and C interfaces to expose the affinity-writing entry points.
• Adds new C++/Fortran unit tests and updates the user guide API documentation.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/unit_tests/f/test_vernier_mod.pf	Adds Fortran unit tests for the new affinity-write routines and output filename behavior.
tests/unit_tests/c++/test_proftests.cpp	Adds C++ tests for affinity write error handling and output filenames.
tests/unit_tests/c++/test_affinity.cpp	New unit tests for affinity-map generation (with mocked syscalls).
tests/unit_tests/c++/CMakeLists.txt	Registers the new test_affinity target.
src/f/vernier_mod.F90	Adds Fortran API bindings/wrappers for affinity map output.
src/c++/vernier.h	Exposes new C++ API methods for affinity writing.
src/c++/vernier.cpp	Implements Vernier affinity-writing entry points and filename construction.
src/c++/mpi_context.h	Extends MPIContext (const-correct getters, on_root(), default tag).
src/c++/mpi_context.cpp	Implements MPIContext const getters and on_root().
src/c++/CMakeLists.txt	Adds affinity.cpp to the library build and installs affinity.h.
src/c++/affinity.h	Declares affinity reporting classes and syscall abstraction.
src/c++/affinity.cpp	Implements affinity map generation and MPI-IO output.
src/c/vernier_c.cpp	Adds C wrappers for the new affinity-write entry points.
documentation/Sphinx/user_guide/api/api.rst	Documents the new affinity-writing APIs for C++ and Fortran.
CONTRIBUTORS.md	Adds a new contributor entry.

Suppressed comments (4)

src/c++/affinity.cpp:89

• core_id is used to index mask without checking bounds. If CPU IDs are sparse or running_on_core() returns an unexpected value, mask.at(core_id) will throw and abort.

    auto core_id = static_cast<std::size_t>(system_calls_->running_on_core());
    char thread_id_char = thread_id_to_char(thread_id);

// If more than one thread is running on the same core, show that with a
// hash symbol.
#pragma omp critical
    {
      if (mask.at(core_id) == '.') {
        mask.at(core_id) = thread_id_char;
      } else {
        mask.at(core_id) = '#';
      }

src/c++/affinity.cpp:148

• The MPI-IO offset for each rank is computed using record_length, but the view datatype and write size use max_record_length. When record lengths differ across ranks (e.g., heterogeneous nodes / different max_cpus), this can cause overlapping writes and a corrupted file.

  // Each rank computes its own offset.
  MPI_Offset my_offset =
      header_length +
      (static_cast<MPI_Offset>(mpi_context.get_rank()) * record_length);

documentation/Sphinx/user_guide/api/api.rst:165

• Grammar: "The tag argument optional" is missing "is".

   communicator. The tag argument optional and, if present, will be appended to

src/c++/vernier.cpp:18

• write_affinity_with_context uses std::stringstream, but <sstream> is not included here. This can fail to compile depending on transitive includes.

#include "vernier.h"
#include "affinity.h"
#include "error_handler.h"
#include "hashvec_handler.h"

#include <cassert>
#include <chrono>
#include <iostream>
#ifdef _OPENMP
#include <omp.h>
#endif

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.