UCX Notified Communication by joe-explr · Pull Request #20 · devreal/ompi

joe-explr · 2026-03-31T20:54:10Z

Signed-off-by: Joseph Antony jajoseph.antony18@gmail.com

devreal

Looks good, just a few comments.

devreal · 2026-04-01T12:27:54Z

+           + (uint64_t)notify * sizeof(uint64_t);
+}
+
+#define CHECK_NOTIFY_IDX(notify)                                              \


I would suggest passing in the module and checking against a value stored in there. OMPI_OSC_UCX_MAX_NOTIFY_COUNTERS is just a crutch until we have a better solution

devreal · 2026-04-01T12:29:05Z

+
+    /* Counters are local memory — just read with a barrier to ensure
+     * any preceding remote writes to this counter are visible. */
+    opal_atomic_rmb();


Not sure this actually does anything. Shouldn't the read barrier come after the counter read to prevent subsequent reads from being reordered?

devreal · 2026-04-01T12:29:49Z

+
+    volatile uint64_t *counter =
+        (volatile uint64_t *)(module->addrs[my_rank] + module->size) + notify;
+    *value = (OMPI_MPI_COUNT_TYPE)opal_atomic_swap_64((volatile int64_t *)counter, 0);


You will need to use UCX to swap the value, otherwise it is not guaranteed to be atomic wrt to other network atomic operations.

devreal · 2026-04-01T12:34:04Z

        /* create the segment */

        size_t total = 0;
+        size_t notify_size = OMPI_OSC_UCX_MAX_NOTIFY_COUNTERS * sizeof(uint64_t);


Replace the use of OMPI_OSC_UCX_MAX_NOTIFY_COUNTERS with a local variable that we can set to something that comes out of the info object. That makes it easier going forward. Also, store the value of that variable in the module (see my earlier comment).

devreal · 2026-04-01T12:35:14Z

Oh, I guess we'll need to get #9 in first?

devreal · 2026-05-20T13:37:36Z

Is this ready to go in?

github-actions · 2026-07-24T20:01:05Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ad1271b: Get and set notify changes

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

This commit adds notification support to the OSC SM component by implementing the put_with_notify, get_with_notify, rput_with_notify, and rget_with_notify functions. These functions perform the same operations as their non-notify counterparts but also increment notification counters after the data transfer completes. The changes include: - Added function pointer types for notify variants in osc.h - Added function prototypes in osc_sm.h - Implemented the notify functions in osc_sm_comm.c - Updated the module template to register the new functions - Removed TODO comments that have been addressed Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

put_with_notify get_with_notify Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

…for a single and multi rank window. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

…eph.antony18@gmail.com>

… <jajoseph.antony18@gmail.com>

…ony18@gmail.com>

MPI-5.1 names the error class for an invalid notification index MPI_ERR_RMA_NOTIFICATION. Rename the placeholder used by the notified RMA work to match the standard. The class was never registered with the error code subsystem, so MPI_Error_string() and MPI_Error_class() did not know about it; add the missing CONSTRUCT_ERRCODE()/OBJ_DESTRUCT() pair in errcode.c. Also fix the binding generator's ERROR_CLASSES list, where the entry was inserted without a trailing comma and so was silently concatenated with the following 'MPI_ERR_TYPE' element rather than added as a class of its own. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Add the two remaining notification-management procedures from MPI-5.1 section 12.6.1: MPI_WIN_SET_NUM_NOTIFY is a blocking, synchronizing collective that sets the number of notification counters attached at the calling MPI process to exactly num_notifications and resets all of them to zero. MPI_WIN_GET_NUM_NOTIFY is local and returns the number of counters attached at target_rank. Both are wired through the osc framework as new module entry points, so components that do not implement them return MPI_ERR_UNSUPPORTED_OPERATION rather than crashing. The osc/sm implementation carves a fixed per-rank counter region out of the shared segment at window creation, which is therefore the effective MPI_WIN_NOTIFICATION_NUM_UB; a request beyond that capacity is rejected with MPI_ERR_ARG. Each rank publishes its own attached count into the shared segment, so the collective needs only a barrier -- no counts have to be exchanged -- and MPI_WIN_GET_NUM_NOTIFY is a plain shared-memory read. Also add the missing put_notify/get_notify entries to interface_profile_sources, which were omitted when those two procedures were introduced. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Three correctness problems in the osc/sm notified communication path: 1. The notification counters were plain uint64_t but are incremented concurrently by remote origins with opal_atomic_add() and polled by the local rank. Type them opal_atomic_int64_t so that the reads in MPI_WIN_GET_NOTIFY_VALUE are atomic and cannot be hoisted out of a caller's polling loop. 2. The notification index was validated *after* the data movement, so an erroneous call had already overwritten the target window (or, for get, the origin buffer) by the time the error was returned. MPI-5.1 section 12.6.1 makes referencing an out-of-range counter erroneous at initiation, so hoist the check above ompi_datatype_sndrcv() in all four notified operations. The check is factored into a helper, which also fixes MPI_GET_NOTIFY returning OMPI_ERR_BAD_PARAM instead of MPI_ERR_RMA_NOTIFICATION. 3. The get paths used opal_atomic_rmb() before incrementing the target's counter. The notification tells the target that the get has read the window, so the constraint is load-before-store, which a load-load fence does not express; opal_atomic_add() is relaxed and adds no ordering of its own. Use a full opal_atomic_mb(). In MPI_WIN_GET_NOTIFY_VALUE the barrier was likewise placed before the counter load, where it ordered nothing; move it after so that it gives the acquire semantics the caller needs. MPI_WIN_RESET_NOTIFY_VALUE also gains the trailing barrier for the same reason. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Adds the six remaining notified operations from the MPI-5.1 draft -- accumulate, get_accumulate, and the four request-based forms -- so that osc/sm covers all eight defined in section 12.3, and sizes the notification counters from window info rather than a compile-time constant. The counter region was a fixed 16 entries per MPI process carved out of the shared segment, and MPI_WIN_SET_NUM_NOTIFY rejected anything larger. That conflicts with the mpi_assert_max_num_notify info key, whose default of 0 the standard defines as "the implementation does not assume any limit on the number of notification counters". The reservation now comes from that key when one is given, and otherwise from a new osc_sm_num_notify_counters MCA parameter. A request beyond the reservation relocates the counters to a dedicated shared segment instead of failing; when the key was given it is a hard bound, since the window was sized on the strength of that assertion. Growth is collective and runs inside MPI_WIN_SET_NUM_NOTIFY, which the standard already defines as a blocking synchronizing collective. Every process agrees on the new layout through an allgather of the requested counts, and on whether the attach succeeded through an allreduce, so a failure at one process cannot leave others incrementing counters that nobody reads. The published count stays clamped to the current allocation until the larger one exists, so a failed growth cannot leave behind a count that would admit writes past the end of the region. A barrier separates the attach from the unlink, because attach opens the backing file by name and the broadcast does not tell rank 0 that the other processes are finished with it. Each process now caches a per-target pointer to the counters, making the lookup on the path of every notified operation a single indexed load -- cheaper than the previous base-plus-offset arithmetic -- so the ability to relocate the region costs the hot path nothing. Also corrects the reset at the end of component_select(), which zeroed the whole node state and so wiped the notification fields it had just written, and removes a stray double semicolon in ompi_osc_sm_fetch_and_op(). Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

MPI-5.1 section 12.2.6, Table 12.1 caches three attributes on every window: MPI_WIN_NOTIFICATION_NUM_SB, the number of notification counters the implementation supports efficiently; MPI_WIN_NOTIFICATION_NUM_UB, the upper bound on that number; and MPI_WIN_NOTIFICATION_VALUE_UB, the upper bound on a counter value. Without them a program has no portable way to ask how many counters it may request, since MPI_WIN_GET_NUM_NOTIFY reports how many are attached rather than how many are available. The values come from a new osc_win_get_notify_bounds entry point on the osc module, queried once per window while it is configured. A component that does not implement notified communication leaves the entry point NULL and all three attributes read zero, which is the honest answer for such a window and is consistent with its notified operations returning MPI_ERR_UNSUPPORTED_OPERATION. For osc/sm the bounds follow the reservation: with an mpi_assert_max_num_notify assertion both NUM_SB and NUM_UB are that value, since the window was sized for exactly it; without one, NUM_SB is what was reserved and nothing bounds NUM_UB short of the notification index type, because the counters grow on demand. The keyvals are appended after MPI_FT so that the existing predefined values stay put, with matching entries in mpif-values.py to keep the C and Fortran numbering identical. The predefined-keyval bitmap is already bounded by MPI_ATTR_PREDEFINED_KEY_MAX and needed no change. Table 12.1 types VALUE_UB as MPI_Count *, and the attribute machinery has no MPI_Count slot -- every other predefined attribute is integer- or address-valued. It is stored as an MPI_Aint, which is the same width wherever Open MPI runs now that 32-bit environments are unsupported; the reasoning is recorded at the call site so the choice does not later read as a type error. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Covers counter management, all eight notified operations in their blocking and request-based forms, notification index errors, growth past the reserved capacity, the mpi_assert_max_num_notify bound, and the three notification window attributes. A second test forces osc/rdma and checks that every notified entry point reports MPI_ERR_UNSUPPORTED_OPERATION without moving any data, and that the attributes read zero. That is the contract which lets components that do not implement the chapter remain untouched, so it is worth testing directly rather than assuming. Both are single-process tests wired into make check, so the shared segment growth path is exercised only in its single-rank form, where the counters are a plain heap allocation. The collective path -- segment creation, broadcast, attach, the status allreduce and the barrier before unlink -- needs a multi-rank test that this harness cannot host. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

The notified communication code carried long block comments that restated the standard at length and recorded design deliberation. Reduce them to short notes that say what the code does and cite the relevant MPI-5.1 section, so the comment density matches the surrounding osc/sm sources. No functional change. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

MPI_WIN_GET_NUM_NOTIFY takes target_rank as a nonnegative integer in the group of the window, but the frontend did not validate it. The osc/sm and osc/ucx backends both range-check it and return MPI_ERR_RANK, so this was not a crash, but argument validation belongs in the frontend under MPI_PARAM_CHECK, consistent with the notified communication operations and with MPI_Win_shared_query. MPI_PROC_NULL is deliberately not accepted here: unlike the notified communication operations, the spec specifies target_rank as nonnegative. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

The base branch renamed the notified-communication error class from MPI_ERR_NOTIFY_IDX to MPI_ERR_RMA_NOTIFICATION; osc/ucx still referenced the old, now-undefined name. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Adds the four accumulate-flavored notified operations, the notification window attributes, and a configurable counter reservation, and corrects MPI_Win_set_num_notify. MPI_Accumulate_notify, MPI_Get_accumulate_notify, MPI_Raccumulate_notify and MPI_Rget_accumulate_notify follow the pattern already used by the put/get variants: issue the base operation, order it ahead of the counter with a fence (or a flush where a result must also be locally valid), then post the atomic increment. The five copies of that increment are now one helper. osc_win_get_notify_bounds was left NULL, so ompi_win_init cached zero for MPI_WIN_NOTIFICATION_NUM_SB, MPI_WIN_NOTIFICATION_NUM_UB and MPI_WIN_NOTIFICATION_VALUE_UB on every UCX window. Zero is how a component says it supports no counters at all, so a program that checked the attributes before calling MPI_Win_set_num_notify would skip notified communication even though UCX implements it. MPI_Win_set_num_notify only ever raised the attached count, but §12.6.1 says MPI_WIN_GET_NUM_NOTIFY returns the value given to MPI_WIN_SET_NUM_NOTIFY, so lowering it has to take effect. Gathering the requested value directly gives that, and also stops an origin from addressing counters the target has since detached. A rank whose argument is out of range no longer returns before the allgather either; that left the rest of the group blocked in a synchronizing collective. The reservation was a hard-coded 16. It is now the osc_ucx_num_notify_counters MCA parameter, overridable per window with the mpi_assert_max_num_notify info key, matching osc/sm. Because the counters share the window's memory registration, which cannot grow, the reservation is a real upper bound and is reported as such in MPI_WIN_NOTIFICATION_NUM_UB; osc/sm can reallocate its segment and so advertises INT_MAX. Ranks agree on one value so the shared-memory layout stays uniform. MPI_Win_notify_threshold (§12.6.3) is still unimplemented. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The counters were appended to the window data and registered as part of the same region. For MPI_WIN_FLAVOR_CREATE the data region is the caller's buffer, sized for the window data alone, so this registered and then wrote past the end of memory the MPI library does not own -- at window creation, and again on every remote notification. Making the reservation configurable turned that from a fixed 128-byte overrun into one the user can scale. It went unnoticed because the notified communication tests only ever call MPI_Win_allocate. The counters now get their own registered region, alongside the window state rather than inside the window data. That removes the overrun, and since the region no longer depends on there being a data region at all, dynamic windows can support notified communication instead of being refused with MPI_ERR_RMA_FLAVOR. The shared-memory segment layout goes back to what it was before notified communication was added. Also from review of the previous commit: MPI_Win_set_num_notify validated its argument on each rank before the allgather, so a rank whose count was out of range returned while the rest of the group stayed blocked in the collective. The check now rides the collective. A refused call also no longer republishes the attached counts, so the group is not left half-reconfigured; the counters are still reset first, which is what makes the standard's "will not return until all processes have adjusted" hold. Window creation had the same defect: a malformed mpi_assert_max_num_notify value returned before the reservation allreduce. The failure is carried through that collective now. The reservation is also range-checked before being narrowed to int for the exchange. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Notification counters are incremented by remote origins with a UCX atomic. Depending on the transport and the atomic mode in use, UCX may emulate that atomic in software on the target's worker rather than offloading it to the NIC. When it does, the counter only advances while the local worker is progressed. MPI_WIN_GET_NOTIFY_VALUE read the counter through a volatile pointer without progressing the worker, so a consumer polling it in a loop -- the natural way to wait for a notification, and the pattern the notified-communication interface exists to support -- could spin forever on a value that can never change. The operation completed only if the application happened to call some other MPI routine that progressed the worker as a side effect. Progress the worker before reading the counter, as every other spin-wait in this component already does. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

MPI_WIN_SET_NUM_NOTIFY is a blocking, synchronizing collective, but its num_notifications argument is local: MPI-5.1 section 12.6.1 states that the number of notification counters "can be different for different MPI processes". Both the C binding and osc/sm validated that argument and returned early, before the osc module's internal allgather. A single rank passing a bad value therefore returned an error while every other rank stayed blocked in that allgather forever, turning an erroneous argument into a hang. The binding rejected negative counts, and osc/sm additionally rejected counts above an mpi_assert_max_num_notify assertion -- a case the binding never covered, so osc/sm could hang even before this change. Drop the range check from the binding and carry each rank's verdict through the collective instead. osc/sm gathers ULONG_MAX as a sentinel that no legal count can collide with, since valid counts come from an int and never exceed INT_MAX; a single-process window has nobody to agree with and still answers immediately. All ranks then see the same gathered array and fail identically, so the window cannot end up half-reconfigured. osc/ucx already carried its verdict through the allgather and needed no change. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

The notification counters were reserved once at window creation and that reservation was a hard cap: MPI_WIN_SET_NUM_NOTIFY rejected any larger count, and MPI_WIN_NOTIFICATION_NUM_UB reported the reservation. MPI-5.1 section 12.2 defines the mpi_assert_max_num_notify info key as an assertion by the caller that it will not request more counters than the given value, and states that when the key is absent (zero) "the implementation does not assume any limit on the number of notification counters". Capping an unasserted window at the default reservation contradicts that, and left osc/ucx unable to satisfy programs that osc/sm -- which grows its counters -- accepts. Grow the counters instead. MPI_WIN_SET_NUM_NOTIFY already gathers every rank's requested count, so all ranks reach the same decision from the same array without extra communication, and grow to the largest request. Growing re-registers the region and re-exchanges base addresses, which is safe precisely because this procedure is a blocking, synchronizing collective: it resets every counter to zero, so a freshly allocated region is already the required contents, and it is erroneous to call it while an access epoch is open, so no remote atomic can be in flight against the region being replaced. The address allgather doubles as the barrier that lets the old region be released. The reservation is never shrunk, so a rank that lowers its count keeps its space and only genuine growth costs a re-registration. A window keeps a hard cap only when *every* rank asserted a bound; a rank that passed no key made no promise. NUM_UB now reports INT_MAX for an unasserted window and the asserted value otherwise, and NUM_SB follows the reservation, so both attributes stay consistent with what MPI_WIN_SET_NUM_NOTIFY will actually accept. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

The request-based notified operations built their request from the underlying rput/rget/accumulate and only then issued the counter update, so the request described the data movement alone. Two consequences. MPI-5.1 section 12.6.4 advises that "completion at the origin entails that the notification counter update has been sent to the target and thus notifications do not rely on progress of decoupled MPI activities at the origin". Because the counter update is a non-fetching atomic issued after the request-bearing flush, MPI_WAIT could return with that update still queued locally, leaving a polling target waiting until the origin happened to re-enter MPI. The failure paths were also malformed: once the underlying operation had succeeded, *request held a live request, yet a failing fence or atomic returned an error. A caller following the usual convention -- on error, do not touch the request -- would then never complete it. The request could not simply be released there either, since UCX already holds it with a completion callback. Issue the data movement and the counter update first, and build the request afterwards. The request is attached to ucp_worker_flush_nb, which covers every operation already issued on the worker regardless of which registration it used, so the flush now covers the counter update as well. Allocating the request last also means every failure point precedes it and those paths return with *request untouched. The accumulate variants get the same treatment. Their underlying accumulate is synchronous -- it ends with a blocking flush and completes its request immediately -- so passing a NULL request runs the accumulate to completion at the target and lets the notification that follows be ordered after it, with a fresh request covering both. Factor the request construction shared by all six operations into osc_ucx_request_over_flush(). rput and rget keep their previous behaviour. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

devreal requested changes Apr 1, 2026

View reviewed changes

joe-explr force-pushed the notified-rma-ucx branch from 93195db to ad1271b Compare July 24, 2026 20:00

joe-explr force-pushed the notified-rma-ucx branch from ad1271b to 2ac49ff Compare July 24, 2026 20:14

devreal and others added 24 commits August 2, 2026 14:02

First draft and TODOs for notified comm

d3c78df

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

osc/sm: Nit picking edits

f1e07c8

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Public APIs for:

06d61ed

put_with_notify get_with_notify Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Edits for Public APIs:

9078470

put_with_notify get_with_notify Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Notified RMA counters memory allocation in the shared memory segment …

26f568e

…for a single and multi rank window. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Editing Notified RMA implementation

650ff8d

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Editing Notified RMA implementation with new design

cf07e67

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Implementing Notify Query

cd5f1b9

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Changes to Notify Query

0c661b2

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Implemented Set Notify Query

45c2eda

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Changes to Notify Query

193c10b

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Configuring memory barries to propagate the update

e47983f

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Reset notify value Implementation Signed-off-by: Joseph Antony <jajos…

095c99d

…eph.antony18@gmail.com>

Addressing review comments and bug fixes Signed-off-by: Joseph Antony…

9d4d06c

… <jajoseph.antony18@gmail.com>

Removing Duplicate changes Signed-off-by: Joseph Antony <jajoseph.ant…

6719fbd

…ony18@gmail.com>

joe-explr added 4 commits August 12, 2026 00:59

UCX Notified Communication

b1707e7

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

UCX Notified Communication changes

4343b91

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

Get and set Notify changes

d0dce0e

Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>

joe-explr force-pushed the notified-rma-ucx branch from 2ac49ff to 8081698 Compare August 12, 2026 04:59

joe-explr and others added 6 commits August 12, 2026 01:27

joe-explr requested a review from devreal August 13, 2026 14:31

devreal force-pushed the notified-rma branch from 17ba85d to bc02964 Compare August 17, 2026 17:28

Navigation Menu

UCX Notified Communication - #20

UCX Notified Communication#20
joe-explr wants to merge 34 commits into
devreal:notified-rmafrom
joe-explr:notified-rma-ucx

joe-explr commented Mar 31, 2026

Uh oh!

devreal left a comment

Uh oh!

devreal Apr 1, 2026

Uh oh!

devreal Apr 1, 2026

Uh oh!

devreal Apr 1, 2026

Uh oh!

devreal Apr 1, 2026

Uh oh!

devreal commented Apr 1, 2026

Uh oh!

devreal commented May 20, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

joe-explr commented Mar 31, 2026

Uh oh!

devreal left a comment

Choose a reason for hiding this comment

Uh oh!

devreal Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

devreal Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

devreal Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

devreal Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

devreal commented Apr 1, 2026

Uh oh!

devreal commented May 20, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants