6.8. The Shared-Memory Datastore: gds/shmem3

This document describes the shared-memory datastore — how a PMIx server builds job, session and modex data inside an mmap’d segment, how every local client reads that same segment in place, how data that cannot be overwritten is nevertheless updated, and why none of it takes a lock. It closes with a step-by-step walk of PMIx_Get, from the public entry point through the pre-computation the library does before it looks anything up, to the read of the shared segment itself, naming each thread shift along the way.

It is a companion to Modex: Exchanging Process Data, which follows a value from PMIx_Put to the aggregated result; this one is about where that result lives. For a contributor-facing orientation inside the code, src/mca/gds/shmem3/AGENTS.md covers the component directory and src/mca/gds/AGENTS.md the framework around it.

6.8.1. The Problem

A node running many processes of the same job holds the same job data in every one of them. With the hash datastore, each client unpacks the job-level blob its server sends and builds its own hash tables from it; the modex is worse, because after a collecting fence every local client holds a full copy of what every process in the job published. On a fat node that is the largest single block of memory PMIx holds, replicated once per rank, and every byte of it was unpacked and stored per rank as well.

shmem3 removes both costs at once. The server builds the actual pmix_list_t, pmix_hash_table_t and pmix_kval_t structures inside a shared-memory segment, and each local client maps that segment at the same virtual address the server used. N clients then share one physical copy, and a client’s “store” of job data is an mmap rather than an unpack-and-rebuild.

Two consequences drive everything else in this document:

• Because the structures are read in place, the two ends share a memory layout, not a wire format. Any disagreement about the size or shape of those types repoints every field a reader touches.

• Because a segment is mapped by processes the writer does not control, a segment a client can see is never written again. Updating data therefore means publishing a new segment, not editing an old one — which is what the chains described below exist for, and what makes lock-free reading possible.

6.8.2. Scope and Roles

• The server is the sole writer. It creates each segment, allocates the structures in it, fills them, and only then makes it readable.

• A client is a reader. It maps segments its server describes to it and reads them; its mapping of the data region is made read-only by the MMU, so this is enforced rather than assumed.

• A PMIx server is also a client of itself — pmix_globals.mypeer has its own datastore module — but that module is pinned to hash, so a server’s own PMIx_Put/PMIx_Get traffic does not go through a segment. What goes through a segment is the data the server holds on behalf of its local clients.

Where the code lives:

File	Contents
src/util/pmix_shmem.[ch]	Segment create/attach/detach/protect, the segment header and its layout stamp, the reference count.
src/util/pmix_vmem.[ch]	Locating a free address range, and reserving one.
src/mca/gds/shmem3/gds_shmem3.h	The trackers, the sm*_data_t structures that live in a segment, the layout ID.
gds_shmem3.c	The TMA allocator, segment placement/creation/attach, the arena, the seg-blob pack/unpack, the module entry points.
gds_shmem3_store.c	Writing job and session data into a segment through the TMA.
gds_shmem3_fetch.c	The read side: pmix_gds_shmem3_fetch() and the chain walks.
gds_shmem3_utils.c/.h	Tracker lookup, the chain publish/head primitives, per-segment status helpers.

6.8.3. When shmem3 Is Selected

Two gates must both pass.

Build gate (configure.m4). The component is compiled only on a 64-bit, non-Apple host. It depends on a large virtual address space and on the /proc/self/maps-based hole finder, so on macOS or a 32-bit target it is not built at all. A change made on a Mac has not been compiled until it has been built on Linux.

Runtime gate (component_query). Even where built, the component disqualifies itself (priority 0) if /proc/self/maps is not accessible. Otherwise it bids priority 20, against hash’s 10, so it wins by default.

assign_module bids 20 normally, 100 when the caller named PMIX_GDS_MODULE="shmem3", and 0 when the caller named some other module. The server hands the selected module’s name to each client it forks in PMIX_GDS_MODULE, so client and server agree by name.

The module deliberately leaves store, assemb_kvs_req and accept_kvs_resp NULL. The framework macros detect the NULL slot on a non-hash module and route those operations to the local server’s own module: shmem3 owns the bulk job/modex data path, and individual put/get traffic falls back. cache_job_info likewise returns PMIX_ERR_NOT_SUPPORTED — unlike hash, shmem3 does not pre-cache, it builds the segment lazily inside register_job_info when the first client connects.

6.8.4. The Name and the Stamp

The version suffix in shmem3 is load-bearing. A client selects its module by name, so a client from an older release that recognizes the name will map the segment — whatever the segment now contains. Since what is shared is a memory layout, a change to pmix_object_t or to anything derived from it silently repoints every field an older peer reads, and the observed failure is a segfault a long way from the change that caused it. So each generation replaces its predecessor: when the layout changes, the number is bumped, older peers no longer recognize the name, and hash — which never disqualifies itself — wins for them. The component has been renamed twice for exactly this reason.

Renaming guards released peers. It cannot guard two builds that both call themselves shmem3 and still disagree, and the most common such case needs no code change at all: --enable-debug adds obj_magic_id to the front of pmix_object_t and two fields to the back, so a debug build and a default build of the same commit disagree by 24 bytes in the base class alone.

Every segment therefore carries a layout stamp in its header:

#define PMIX_GDS_SHMEM3_LAYOUT_ID                                     \
    ((uint32_t)(                                                      \
        PMIX_GDS_SHMEM3_LAYOUT_VERSION                                \
      +   3u * (uint32_t)sizeof(pmix_object_t)                        \
      +   5u * (uint32_t)sizeof(pmix_list_t)                          \
      /* ... one term per type that lives in a segment ... */         \
    ))

It is computed, not maintained — a hand-maintained version number is one somebody has to remember to bump — and the distinct prime multipliers keep a growth in one type from cancelling a shrink in another. pmix_shmem_segment_attach() refuses a segment whose stamp differs, returning PMIX_ERR_NOT_SUPPORTED; shmem3_attach() turns that into PMIX_ERR_TAKE_NEXT_OPTION and the client quietly uses hash.

The header that holds the stamp is fixed-width and free of any class-derived type or debug conditional, on purpose: a stamp that can itself shift cannot detect a shift.

6.8.5. Anatomy of a Segment

A segment is a file under the session/nspace tmpdir, mapped MAP_SHARED. Its layout:

+--------------------------------------------------+  <- hdr_address
| pmix_shmem_header_t                              |
|   ref_count (atomic), magic, layout_id           |
|   ... padded out to a page boundary ...          |
+--------------------------------------------------+  <- data_address
| shared_{job,session,modex}_data_t                |
|   tma            (the allocator's state)         |
|   current_addr   (the bump pointer)              |
|   <the roots: lists, hash table, key index>      |
+--------------------------------------------------+
| everything those roots point at, carved out      |
| of the same segment by the bump allocator        |
|                                                  |
|                        ...                       |
+--------------------------------------------------+

The header is deliberately outside the data region. It carries the reference count that attach and detach maintain, which is what lets one generation of a segment be handed off while readers are still on it: the backing file survives until the last holder lets go, and dropping the last reference unlinks it.

Important

The size a client must attach with is the mapped footprint — the value the creator’s handle carried after pmix_shmem_segment_create() returned, which is pmix_shmem_utils_segment_footprint() of the requested data size, not the requested size. Pass the smaller number and the mapping ends a page short of the data region, which nothing can detect until a reader faults on the tail. That is why the creator’s shmem->size goes on the wire.

6.8.5.1. The allocator

Everything in a segment is allocated by a bump allocator, the pmix_tma_t (“temporary memory allocator”) wired up in gds_shmem3.c. PMIX_NEW(type, tma) and the pmix_tma_* routines carve from current_addr and advance it, 8-byte aligned; free is a no-op.

Two properties are easy to undo by accident:

• Every block carries a 16-byte header immediately ahead of the address handed out — the extent plus a magic. Its only consumer is tma_realloc(), which needs the old size to know how much to copy. Nothing in the segment is reached by walking allocations, so this is not part of the shared layout. It must stay a multiple of 8, or every object in the segment is misaligned. It replaced a side hash table mapping address to extent, which cost two heap allocations and a pointer-keyed hash insert on every allocation and only ever grew.

• Nothing zeroes a block, and nothing needs to. A bump allocator only hands out space no caller has touched, and pmix_shmem_segment_create() opens its backing file O_CREAT | O_TRUNC and ftruncates it from empty — so every page of a fresh segment reads as zero. The O_TRUNC is load-bearing rather than tidiness: a backing path collides only with a file some earlier server left behind when it died, and paths are built from pids, which get reused.

Overflowing a segment abort()s the process with a guidance message. There is no growing: the pre-sizing arithmetic plus the segment_size_multiplier MCA parameter is the only guard, so segments are sized generously.

Warning

Never PMIX_CONSTRUCT an object destined for shared memory. PMIX_CONSTRUCT records the constructing process’s addresses into the object; shared objects must be built through the TMA (PMIX_NEW(type, tma)) so their addresses live in the segment. This is the single easiest way to corrupt the store.

6.8.5.2. The three kinds of segment

ID	Root structure	Holds
JOB	shared_job_data_t	local_hashtab (all job-level and per-rank job data), nodeinfo and appinfo lists, and the segment’s key index.
SESSION	shared_session_data_t	sessioninfo and nodeinfo lists for the session. Shared between every job in that session.
MODEX	shared_modex_data_t	hashtab holding one collecting fence’s aggregated result, and the segment’s key index.

Each is placed, created and attached by shmem3_segment_create_and_attach(), and each carries its own pmix_tma_t: an allocation is always made against the segment the object will live in.

6.8.5.3. Hash tables here are keyed by rank

Both local_hashtab and the modex hashtab hold one element per rank. pmix_hash_store() looks up a single pmix_proc_data_t for the rank and hangs that rank’s values off it in a pointer array; the values are not table elements. Job-level keys all share the one element belonging to PMIX_RANK_WILDCARD.

That makes two different numbers matter when pre-sizing a segment, and they are not interchangeable — the element count sizes the table and the per-rank structures, while the key/value pair count sizes the stored values and the key index. Feeding the pair count into the table is the original bug in this arithmetic: a 32-rank job built a table with tens of thousands of elements.

The sizing code asks the code that spends the memory rather than restating it. Four entry points exist for that, each living beside what it describes:

Ask	For
pmix_hash_table_sizeof_storage(n)	the element array pmix_hash_table_init(ht, n) allocates
pmix_hash_sizeof_proc_storage()	one rank’s per-proc object and its two pointer arrays
pmix_keyindex_sizeof_storage(nkeys)	a key index sized for nkeys
pmix_hash_sizeof_key_entry(len)	one newly registered key

Do not open-code any of them here. And whatever is put in a segment must be in that segment’s estimate before it is put there: the allocator cannot grow, so an index that rehashes runs off the end of the segment and aborts the server partway through register_job_info() — which presents as every client hanging forever for job data that is never coming, not as a crash on the process that actually failed.

6.8.6. Every segment carries its own key index

Inside the datastore a key is not a string but an integer (see Modex: Exchanging Process Data, “The Key Index”). In a shared segment that integer crosses a process boundary: the server writes it, every local client reads it. The two ends agree in two different ways, and only one of them needs anything stored in the segment.

• Reserved attributes agree by construction. Their ids come from the generated dictionary and are pinned in contrib/dictionary_ids.txt precisely so that they are the same in every process and every release. Nothing has to be carried, so nothing is: lookup_key() resolves any id below PMIX_INDEX_BOUNDARY against the process-global index instead.

• Non-reserved keys do not agree. A key the Standard does not define is numbered on first encounter, in an order that depends on what arrived and when. Those go in the segment’s own index, beside the data they describe, where they only have to be consistent within that one segment — which they are, because exactly one process writes it.

So job->smdata->keyindex and job->smmodex->keyindex are allocated in their segments through the segment’s TMA and hold the non-reserved keys only. A private index numbers from PMIX_INDEX_BOUNDARY upward, so the id itself says which half it came from.

Important

A client maps the segment read-only, so a reader must reach these only through the non-registering lookups — pmix_hash_find_key(), or pmix_hash_lookup_key() with a known index. Registering on a lookup would mean a reader writing into the segment, and a key that was never stored cannot be found anyway.

This replaces an earlier scheme in which the server shipped its whole key index in the job-info reply and the client renumbered its own global index to match. Putting the index next to the data removes the requirement rather than satisfying it; there is no key-index blob on the wire any more, and it should not come back. If you find yourself needing to reconcile two dictionaries across a process boundary, put the index next to the data instead.

6.8.7. The Fixed-Address Contract

In-segment pointers are stored raw. They are valid in a client only because the client maps the segment at the same virtual address the server used, so no fix-up is needed on attach. The server packs, per segment, a seg blob naming the backing path, the mapped size, and the header address, and the client maps there.

Choosing an address that a different process can also obtain is the hard part.

6.8.7.1. Placement

pmix_vmem_find_hole() parses /proc/self/maps and locates the biggest hole in the address space. Two refinements matter:

• Not the midpoint. VMEM_HOLE_BIGGEST aims at the middle of the biggest hole, and hwloc and Open MPI — which share this routine’s ancestry — aim there too. That convergence is exactly what makes an address picked in one process likely to be taken in another. VMEM_HOLE_BIGGEST_OFFSET (the offset_placement parameter, on by default) lands a quarter of the way in instead: far from the crowded midpoint, far from the populated region near the binary, and still deep inside tens of terabytes of empty space. Placing at the top of the hole was tried first and failed immediately — the top of that hole is the underside of the executable’s load address, where ASLR moves things most and where the heap grows.

• Scattered by namespace. Deterministic placement is what lets a server and its clients agree on an address without exchanging one, but it also makes two things that have no reason to agree land on top of each other — concurrent jobs on a node being exactly that. The address is therefore displaced within a window by a scatter value derived from hashing the namespace (FNV-1a): identical for every process of one job, different between jobs. The independent-placement path mixes in the segment id and the retry count as well, so a job’s three segments are not all aimed at one address and a retry does not keep naming the address it just lost.

The scan itself is not the expensive part, and this is recorded so the suspicion is not re-derived: measured cost is about 0.17 µs per VMA, essentially linear (16 VMAs → 5 µs; 2016 → 345 µs). A PRRTE daemon carries about 72 VMAs, so the scan costs it roughly 15 µs, under 1% of the per-segment cost. Caching a hole would trade that for a stale-address failure mode on a path whose whole correctness rests on client and server agreeing about an address.

6.8.7.2. The address-space arena

A client attaches the job segment during PMIx_Init, when its address space is nearly empty. It attaches a modex segment at a fence, by which time it has loaded an MPI stack, opened components, grown arenas and registered fabric memory. The server picks both addresses from its own map. The first attach almost always works; the second is a gamble, and openpmix#4156 was that gamble lost — about one run in 25 on a four-node job, taking every rank on a node at once.

The fix is to stop asking. At register_job_info time the server calls arena_reserve(), which claims one contiguous range — the job segment’s footprint plus one slot per modex generation that can be live at once — using pmix_vmem_reserve(), an inaccessible PROT_NONE mapping that commits no memory and costs one VMA. The range travels to clients on every seg blob, and each client claims the same range inside PMIx_Init, the one moment it reliably can. Every later mapping is then MAP_FIXED over ground the process already holds (PMIX_SHMEM_MAP_OVER_RESERVATION), which cannot lose a race.

Four things about it are load-bearing:

• Carve by the footprint, not the requested size. A segment maps a page of header ahead of its data. Carving to the requested size overlapped each segment with the next by one page, and the next MAP_FIXED silently replaced it.

• Detach restores the reservation. pmix_shmem_segment_detach() re-maps PROT_NONE over an in-arena range rather than unmapping it, or the first released segment would punch a hole in the arena for something else to take.

• A modex generation holds its slot for as long as it is readable, which is not just until the next one arrives. arena_alloc_modex() therefore reads occupancy off the live segments — the build slot plus every published generation — and takes the lowest free slot. Deriving it rather than keeping a tally means no path that releases a generation can leave the accounting stale.

• The session segment is deliberately outside the arena. A session’s segment is held by jobs other than the one that built it, while job_destruct() unmaps the arena wholesale, so a session placed inside it would go away under a live holder.

If the arena cannot be reserved, or a generation does not fit its slot, every segment places itself independently exactly as before — degraded, never broken.

6.8.7.3. When an attach fails

shmem3_attach() turns “could not map at the required address” and “layout stamp does not match” alike into ``PMIX_ERR_TAKE_NEXT_OPTION``. That propagates up unchanged and triggers the framework’s GDS fallback (fallback_to_next_gds() in src/client/pmix_client.c): the client switches to hash and re-requests its job data with PMIX_GDS_FALLBACK_CMD. A failed attach is therefore not fatal to PMIx_Init.

That fallback exists only at init, and the distinction is carried explicitly rather than inferred. PMIx_Init can fall back because it re-requests its job data and the server can re-register it in another module’s format. Nothing that arrives afterwards can: a completed modex cannot be re-delivered in hash format without a new wire command, and an update to job or session data arrives on a one-way notification with no re-request behind it at all.

So each of the three deliveries into client_connect_to_shmem3_from_buffi() names which kind it is — PMIX_GDS_SHMEM3_ATTACH_INIT for the job-info reply, PMIX_GDS_SHMEM3_ATTACH_UPDATE for a modex generation or an update — and the failure path keys off that:

• INIT drops the job tracker and lets PMIX_ERR_TAKE_NEXT_OPTION propagate, because nothing will read that tracker again.

• UPDATE clears only the failed segment’s status, keeps everything the client has been reading since PMIx_Init, and swallows the status. The client misses on whatever that one segment held, and the miss goes up to the server like any other.

That test used to be on the segment id — sparing the modex and nothing else — so a session update that failed to map took the whole job tracker down with it, and with it the job and session segments the client had been reading since PMIx_Init. That is openpmix#4156 reached by a different delivery.

test/unit/update_attach_fail holds it, and has to ask with PMIX_OPTIONAL to do so: an ordinary PMIx_Get misses locally, goes up to the server and is answered correctly, so the only visible cost is a round trip per lookup — which is exactly how a defect like this survives.

Keeping the tracker is only half of it, though, and the other half is not a slower answer but a wrong one. An update publishes a segment carrying the values that changed, and job_fetch() stops at the newest segment holding the key. A client that could not map that segment therefore does not miss: the older segment is still on its chain and still answers, with the value the new one was published to replace, PMIX_SUCCESS, for the rest of the run — while every peer that did map it reads the new value. So a refused delivery marks that realm chain_incomplete and the realm declines locally until a delivery completes with nothing refused. Declining turns the silent wrong answer back into the miss the paragraph above describes, and the server answers it correctly.

One caller sees that as a failure rather than a round trip, and it is worth knowing which. PMIX_OPTIONAL says “do not ask the server”, so a request carrying it against a realm that is declining has nowhere to be answered from and returns PMIX_ERR_NOT_FOUND. That is the honest answer — the process genuinely cannot answer without asking — but a caller that uses PMIX_OPTIONAL as a cheap presence test will see a key it had before appear to vanish, until a delivery completes with nothing refused and the realm answers locally again.

6.8.7.4. Clients are read-only, and the MMU enforces it

As the last step of attaching, a client calls pmix_shmem_segment_protect_data() and drops write access to the data region. A client-side write is then a SIGSEGV on the instruction that did it, rather than corruption some other process trips over later.

The internal header stays writable on purpose: it holds the reference count that attach and detach maintain, and a reader that could not write it could not let go of the segment. That is why the protection starts at the data region, a page-aligned header into the mapping — and why the geometry lives in pmix_shmem.c rather than at the call site.

Getting here required removing two writes a reader was making: a hash-table lookup that recorded the table’s key type, and the job segment’s indices being minted against the process-global key index that the client then rewrote to match the server’s. Both are gone. Do not add a client-side write path; it will not silently work.

6.8.8. How Data Is Written

6.8.8.1. Job and session data

The first client of a namespace to connect drives register_job_info, and the server does the following, all on the progress thread:

1. fetch_local_job_data() asks for a complete copy of the job-level information the host registered.

2. get_local_job_data_info() packs it to measure it, producing both the element count and the key/value count discussed above.

3. prepare_shmem3_stores_for_local_job_data() reserves the arena and creates the job (and, if the job named a session, session) segments at the computed sizes.

4. pmix_gds_shmem3_store_local_job_data_in_shmem3() walks the fetched list and writes it through the segment’s TMA. It recognizes PMIX_APP_INFO_ARRAY, PMIX_NODE_INFO_ARRAY, PMIX_PROC_INFO_ARRAY and PMIX_SESSION_INFO_ARRAY and routes each to its own store; anything else is a plain job-level key stored into local_hashtab under PMIX_RANK_WILDCARD, against the segment’s own key index.

5. The segments are marked READY_FOR_USE and then published (below). The job’s goes first: a client told about the session before the job would find a job tracker with nothing in it.

6. cache_connection_info_for_job_shmem3() packs job->conni — the namespace name, a seg blob per published job segment, the tombstone list, and a seg blob per published session segment. That buffer is cached and copied into the reply for every later client, so the work above happens once per namespace.

On the client, store_job_info is simply client_connect_to_shmem3_from_buffi(): unpack each seg blob, attach if not already attached, publish locally. That runs on the client’s progress thread during PMIx_Init.

6.8.8.2. Modex data

server_store_modex delegates to the base envelope walker (pmix_gds_base_store_modex) with server_store_modex_cb() as the per-proc callback. Each modex gets a new segment, one per generation:

• An empty build slot means this blob starts a generation. If a generation is already published, the counter advances first — it names the backing file (modexdata.<generation>), so successive generations do not collide, and it dates tombstones.

• get_modex_sizing_data() estimates the segment from the blob, the segment is created and attached, and modex_smdata_construct() builds the hash table and key index inside it.

• Each unpacked pmix_kval_t is stored into that table against that segment’s key index.

• A NULL blob means the walker is done: the generation is marked READY_FOR_USE and published.

A finished generation is never written again. Writing a second modex into the first one’s segment is not a latent problem but a prompt one: the segment was sized from the payload that created it and the allocator behind it cannot grow, so a larger second modex overran it and aborted the server — taking the daemon, and the job, with it. examples/modex_twice.c reproduces that from four nodes up.

The handoff is safe because the backing file is reference counted: dropping the server’s handle leaves a client that still has it mapped with a valid mapping, and the file survives until the last holder lets go.

Note

server_store_modex_cb() must return PMIX_SUCCESS for a proc blob it consumed. Its natural exit is the unpack end-of-buffer code, and the base envelope walker reads any non-success return as failure of the whole server contribution — it stops, and then converts the same code to success for its own caller. Returning it would store the first proc of each contribution, discard the rest, and report success.

6.8.9. How Data Is Updated

Since a published segment can never be rewritten, an update is a new segment carrying only what changed, published at the head of a chain. A read walks the chain newest-first and stops at the first segment that answers.

6.8.9.1. The chain

typedef struct pmix_gds_shmem3_seg_t {
    struct pmix_gds_shmem3_seg_t *prior;   /* written once, never again */
    pmix_gds_shmem3_job_shmem3_id_t smid;
    pmix_gds_shmem3_status_t status;
    pmix_shmem_t *shmem3;
    void *smdata;                          /* the shared_*_data_t root */
    uint32_t generation;
    bool is_delta;
} pmix_gds_shmem3_seg_t;

typedef _Atomic(pmix_gds_shmem3_seg_t *) pmix_gds_shmem3_chain_t;

It is deliberately not a pmix_list_t, and that difference is the whole point:

• a node is filled in completely before it is published, is never written again, and is never removed;

• publishing is therefore one release-store of the head (pmix_gds_shmem3_chain_publish()) and entering it is one acquire-load (pmix_gds_shmem3_chain_head());

• so a reader that has the head walks a chain that cannot change beneath it, on any thread, with no lock.

A pmix_list_t cannot do that: it is doubly linked through a sentinel and carries a length counter, so a prepend is several stores and a walker can catch any of them. None of the list operations are wanted anyway — the chain is only ever prepended to and walked one way — which is why a node carries a bare ->prior rather than deriving from pmix_list_item_t. A node is a plain allocation rather than a pmix_object_t, because a refcount would suggest it can be released from more than one place, which is precisely what a lock-free chain must not permit.

All three kinds of data are chains, and they behave identically:

Chain	Published by	Grows when
job_chain	register_job_info(), server_add_job_data()	a host adds to an already-registered job’s data
session->segments	the first job in the session	the session’s description changes
modex_chain	each collecting fence	every fence

In each case the build slot on the tracker (job->shmem3/smdata, job->modex_shmem3/smmodex, sesh->shmem3/smdata) holds the segment being written and no reader touches it; publishing onto the chain is what makes it readable, and it empties the slot. So there is no separate “current” for a reader to consult first, and no window in which a reader could catch a segment half-built or half-retired.

That is also why the build fields are not the place to ask whether there is data, or which segment to describe to a client: both questions are answered from the chain head.

Nothing removes from a chain. The only thing that ever takes nodes away is pmix_gds_shmem3_chain_destruct() in the job tracker’s destructor, which runs when the last reference to the tracker has gone and therefore has no reader to race. Keeping a superseded generation is redundant rather than wrong — the walk stops at the newest segment holding the key. Redundant costs address space; unmapping under a reader costs correctness.

6.8.9.2. What clients are told

• The job and session chains are packed oldest first (pack_seg_chain(), recursively). Order is load-bearing: the client publishes each segment as it attaches it, so describing them oldest first leaves its chain in the same order as the server’s, newest at the head.

• The modex ships only its newest generation. A client already holding older generations keeps them; one attaching now is told about the rest at its next fence.

• A running client learns of a new job segment through pmix_server_notify_gds_update() and of a session update through server_pack_update/client_accept_update — all of which funnel into the same unpacker, whose duplicate check is what makes an update idempotent.

6.8.9.3. Delta modex generations

The flag byte in the modex envelope says whether a contribution is cumulative (it repeats everything its processes have published) or delta (it repeats nothing). A delta generation is not self-contained: the generations before it are still the only copy of everything it did not repeat, so they must stay mapped and answerable. That is why the modex is a chain and not a single current segment, and why the arena allocates slots rather than alternating between two.

modex_fetch() accordingly walks newest-first. A keyed lookup stops at the newest generation holding the key. A NULL-key lookup — “everything this process published” — consults every generation and drops a copy of a key a newer one already supplied, or the caller sees that key twice with the stale value second.

The chain is empty of prior generations unless a delta has been stored, so the ordinary case is the single lookup it has always been. examples/modex_twice.c is the canary, and contrib/dockerswarm/run-gds-tests.sh drives it twice: once cumulatively, once with pmix_server_fence_delta_modex=1.

6.8.9.4. Deletion: tombstones

This component cannot take a key out — the data is in a segment local clients have mapped. del_key records that the key is gone and every read consults the record.

The record is process-local, on job->tombstones, and it is a chain with the same discipline as the segment chain: a node is complete before it is published, never rewritten, never removed until the tracker dies. So recording a deletion and reading one both run without a lock. A key deleted again at a later generation publishes a second node rather than updating the first; tombstoned() is satisfied by any matching entry, so the newer node says everything an update would have. A repeat that would not change the answer is skipped, which keeps the chain bounded.

Putting tombstones in shared memory would mean a new segment, mapped by every local client, for a few bytes per deleted key — and it would not save the step that matters, because a client attaching afterwards has to be told either way. Each process builds its own from the notification its server sends, and pack_tombstones() adds the list to the cached job-info reply for later arrivals. del_key() drops job->conni for exactly that reason: the cached reply still says the key exists.

A tombstone carries the modex generation it was recorded at. Job data is written once and never re-published, so job_fetch() asks with UINT32_MAX and a tombstone against it always applies. Modex data can legitimately come back, so modex_fetch() asks with each generation’s own number as it walks, and a tombstone shadows only generations up to the one it was made at.

Both ends advance job->modex_generation, and only the server’s names anything. It was introduced as a server-side counter naming the next backing file, but it is also what dates a tombstone, and del_key() runs on the client too. A client that left it at zero stamped every tombstone with generation zero, so generation <= t->generation held for every generation it ever mapped and the removal never expired there. The two counters are never compared with each other — each only has to advance whenever its process takes on a new generation.

6.8.10. How Locks Are Avoided

pmix_shmem3_module.is_tsafe is true, and that is not decorative. try_local_fetch() tests it on every keyed PMIx_Get — so on a client, pmix_gds_shmem3_fetch() normally runs on whatever thread called PMIx_Get, while the progress thread carries on underneath it.

There is no lock on that path, and that is the design. A lock here is paid by every PMIx_Get, and a library pulling thousands of values during its init pays it thousands of times — which is precisely what this component exists to avoid.

What makes it safe is that everything a read walks is one of three things:

What a read walks	Why it needs nothing
the contents of a segment	written once, before any client can see it, and never again
the segment itself	never unmapped while a reader holds the tracker — a mapping cannot be withdrawn mid-read
the chains (job, session, modex, tombstones)	a node is complete before it is published, never rewritten, never removed

Entering a chain is a single acquire-load paired with the release-store that published its head; the ->prior pointers are immutable, so the rest of the walk is ordinary pointer chasing.

The synchronization that does exist is exactly three things:

1. The component’s ``joblock``. It guards the spine of the job list, because a reader that is not on the progress thread could otherwise have the list mutated under it — pmix_list_remove_item() is several stores, and a walker can catch any of them. It is held across exactly two things and dropped: the walk itself, and the PMIX_RETAIN that claims what the walk found, so a tracker cannot be removed in the gap between finding it and owning it. It does not cover reading a tracker’s contents. Every site that appends to or removes from the list takes it, and every one of them releases its own reference outside the lock — if a reader holds one, that release is not the last, and the destructor that detaches the segments does not run until the reader is done.

2. The reference a reader holds on the job tracker. pmix_gds_shmem3_acquire_job_tracker() takes it under the lock and the reader releases it when finished. A tracker owns the mappings of its segments, so releasing the last reference detaches them; holding one means the progress thread can deregister the nspace underneath the reader and the memory being walked still stays put. The refcount is a C11 atomic, so this works across threads.

3. The atomics themselves: the chain heads, the tombstone head, and job->modex_generation (an pmix_atomic_uint32_t, because a reader on another thread compares a tombstone against it).

A job->datalock used to stand here, and it was needed because none of the three rows above were true: a retired generation could be unmapped under a reader, the current generation was moved between two fields in several stores, and the tombstone list was an ordinary pmix_list_t being appended to. Each was fixed in turn rather than locked around.

The component’s sessions list has no lock, and the reason is an invariant rather than an oversight: it is only ever walked from the progress thread.

That is not because a session-realm get is served somewhere else. It is served right here, out of the session segments, by fetch_sessioninfo(). What differs is the thread it arrives on. try_local_fetch() is a short circuit onto the caller’s own thread, and declining it does not route the request away from shmem3 — it means the ordinary PMIX_THREADSHIFT(cb, get_data) path runs instead and the same pmix_gds_shmem3_fetch() is called from the progress thread. A realm request is declined because resolving one is not a plain lookup: get_data() may first have to fetch another proc’s session id and rebuild the info array around it, which is progress-thread work.

Both ways into fetch_sessioninfo() are covered. A PMIX_SESSION_INFO qualifier sets lg->sessioninfo and lg->sessiondirective; a session-classified key sets one of the three realm flags; and the whole-job read that also lands there has a NULL key. Every one of those is in the decline list. Note this does not depend on the two sides classifying a key in the same order — they do not, process_request() tries node/app/session and shmem3_fetch_from_job() tries session/node/app — because all three realm flags decline, so a key landing in any realm declines whichever one it landed in.

pmix_gds_shmem3_get_session_tracker() also writes job->session on its way through, so an application thread arriving there would be a race on the tracker, not only on the list spine. Relaxing that gating means giving this list the same treatment joblock gives the other one.

Important

If you add state that a read consults, make it immutable or make it a chain. Do not reach for a mutex: a lock on this path is paid by every PMIx_Get.

6.8.11. PMIx_Get, Step by Step

This section follows a single PMIx_Get on a client. The code is src/client/pmix_client_get.c and, for the datastore half, src/mca/gds/shmem3/gds_shmem3_fetch.c.

6.8.11.1. Step 0: entry checks

PMIx_Get rejects, in order: an uninitialized library (PMIX_ERR_INIT), a NULL val (there is no way to hand back an answer), a stopped progress thread (PMIX_ERR_NOT_AVAILABLE), and a key longer than PMIX_MAX_KEYLEN. Nothing shared has been touched yet; all of this is on the caller’s thread.

6.8.11.2. Step 1: pre-computation — the lg struct

lg = PMIX_NEW(pmix_get_logic_t) allocates the object that carries everything the request means, as opposed to what the caller literally wrote. process_request() fills it in. This is the whole of the library’s interpretation of the request, and it happens before any thread shift, because it touches nothing shared.

typedef struct {
    pmix_object_t super;
    pmix_proc_t p;            /* the resolved target */
    bool pntrval;             /* PMIX_GET_POINTER_VALUES */
    bool stval;               /* PMIX_GET_STATIC_VALUES  */
    bool optional;            /* PMIX_OPTIONAL           */
    bool immediate;           /* PMIX_IMMEDIATE (server acts on it) */
    bool refresh_cache;       /* PMIX_GET_REFRESH_CACHE  */
    pmix_scope_t scope;       /* PMIX_DATA_SCOPE         */
    bool sessioninfo, sessiondirective;  uint32_t sessionid;
    bool nodeinfo,    nodedirective;     char *hostname; uint32_t nodeid;
    bool appinfo,     appdirective;      uint32_t appnum;
} pmix_get_logic_t;

process_request() does six distinct things:

a. Reject the two unusable combinations. Both proc and key NULL is PMIX_ERR_BAD_PARAM; so is a NULL key with PMIX_RANK_WILDCARD, since “all data from every rank” has no answer.

b. Infer the realm from the key. pmix_check_node_info(key), pmix_check_app_info(key) and pmix_check_session_info(key) set lg->nodeinfo / appinfo / sessioninfo. A key like PMIX_NUM_SLOTS is a node-realm question whether or not the caller said so.

c. Walk the caller’s info array. Each recognized qualifier lands in lg. Typed qualifiers are validated here rather than trusted: PMIX_DATA_SCOPE must really carry a PMIX_SCOPE, PMIX_HOSTNAME a non-NULL string (and it is strdup‘d, because lg outlives the call and the caller’s array is not ours), and PMIX_NODEID / PMIX_APPNUM / PMIX_SESSION_ID go through PMIx_Value_get_number(). The explicit realm directives (PMIX_JOB_INFO, PMIX_NODE_INFO, PMIX_APP_INFO, PMIX_SESSION_INFO) override the inference in (b) and set the matching *directive flag, which later distinguishes “the caller asked for this realm” from “we deduced it”.

d. Answer the requests that need no datastore at all. Three keys are served straight from pmix_globals: PMIX_PROCID (with a NULL proc), PMIX_VERSION_NUMERIC, and PMIX_RANK when the caller passed our own nspace and PMIX_RANK_INVALID. Each returns PMIX_OPERATION_SUCCEEDED with *val already prepared, honoring lg->stval (copy into the caller’s storage) and lg->pntrval (hand back &pmix_globals.myidval / myrankval). These never post an event and touch nothing shared, which is why the progress-thread check in step 2 sits below this point — they remain usable from inside an event handler.

e. Resolve the target into ``lg->p``. A NULL or empty nspace means our own. A NULL proc means the key is expected to be globally unique, which resolves to PMIX_RANK_UNDEF — except when a node or app realm was selected, where “our own” is the sensible referent and our rank is used instead.

f. Translate a group. If the caller named a group in proc->nspace, pmix_client_convert_group_procs() maps it to the real proc, and exactly one proc must come back.

6.8.11.3. Step 2: refuse to deadlock

pmix_progress_thread_check_blocking("PMIx_Get") returns true if the caller is already standing in the progress thread. Everything below either blocks on that thread or hands work to it and waits, so such a caller would be waiting for itself; PMIX_ERR_WOULD_BLOCK is returned instead of hanging.

6.8.11.4. Step 3: optional cache refresh — a blocking round trip

If lg->refresh_cache is set, refresh_cache(&lg->p) packs a PMIX_REFRESH_CACHE command, sends it, and PMIX_WAIT_THREADs on the reply — on the caller’s thread. The reply is unpacked by refcb() on the progress thread (a PTL receive callback) and stored into pmix_globals.mypeer’s module. Note that PMIx_Get_nb blocks here too, which is why it carries the same progress-thread guard scoped to this branch.

6.8.11.5. Step 4: the caddy

cb = PMIX_NEW(pmix_cb_t), carrying lg, the caller’s key and info/ninfo by pointer (the caller guarantees they stay valid), and — for the blocking form — cb->cbfunc.valuefn = _value_cbfunc with cb->cbdata = cb, so completion sets cb->status/cb->value and wakes cb->lock. The non-blocking form stores the caller’s callback instead and sets cb->checked = true to route completion through gcbfn().

6.8.11.6. Step 5: the local short circuit — no thread shift at all

try_local_fetch(cb, lg) answers the request on the caller’s own thread when the datastore holding the answer can be read from one. This is where shmem3 earns its place: the thread shift is the dominant cost of a get that hits locally — measured at roughly 4.9 µs against about 130 ns of actual lookup underneath it.

The gating is deliberately narrow. It declines if:

• pmix_client_globals.fast_get is off — that is, the developer switch PMIX_GET_ON_PROGRESS_THREAD is set in the environment (see below);

• the key is NULL — “everything this proc put” is an aggregate across scopes, not a lookup;

• any realm redirection is in play (nodeinfo/appinfo/ sessioninfo, any *directive, a hostname, a nodeid) — resolving those takes further fetches and an info-array rebuild that belong on the progress thread;

• refresh_cache was asked for;

• this is a singleton, or we are not connected;

• PMIX_GDS_FETCH_IS_TSAFE(pmix_client_globals.myserver) says no.

That last check is re-resolved every time rather than cached: fallback_to_next_gds() can re-point this peer at a different module at run time, and a stale answer would be a read of a store this process has abandoned.

On a pass it sets cb->proc = &lg->p, cb->scope = lg->scope, calls PMIX_GDS_FETCH_KV — which lands in pmix_gds_shmem3_fetch(), described in step 8, running on the caller’s thread — and shapes the result with process_values(). A miss, or a process_values() that declines, drains cb->kvs and falls through to the ordinary path. This is a short circuit, never a partial one: the drain matters because process_values() distinguishes “the value” from “an aggregate of everything this proc put” by counting that list, so one entry left behind by a failed fetch would turn a scalar get into a data array.

Thread shift 1

PMIX_THREADSHIFT(cb, get_data) — taken only when the short circuit above declines. The blocking form then does PMIX_WAIT_THREAD(&cb->lock); the non-blocking form returns immediately.

6.8.11.7. Step 6: get_data() on the progress thread

Everything from here runs on the progress thread.

Realm resolution. If lg->nodeinfo, lg->appinfo or lg->sessioninfo is set, the request first has to be given the identity of the node, app or session it is about. Each branch:

• uses what we already know if the target is us (pmix_globals.hostname, nodeid, appnum, sessionid);

• otherwise issues a small PMIX_OPTIONAL sub-fetch on a stack caddy for PMIX_HOSTNAME / PMIX_NODEID / PMIX_APPNUM / PMIX_SESSION_ID — against pmix_client_globals.myserver’s module on a client, pmix_globals.mypeer’s on a server, because a client’s job-level data is stored against the server’s module;

• short-circuits when the identifier is what was asked for (a get of PMIX_HOSTNAME in the node realm is answered right there);

• otherwise copies the caller’s directives into a fresh array with room for two or three more (copy_directives()), appends the realm directive if it was inferred rather than given, the identifier, and PMIX_OPTIONAL, and sets cb->infocopy. That array is what _pack_get() will put on the wire.

The transfer status matters here and is checked: PMIx_Info_xfer() sets the destination’s type before it can fail, so a directive that would not copy leaves a slot naming a type with nothing behind it — not an empty slot the server can skip.

The local lookups (label ``doget``). In order:

1. PMIX_GDS_FETCH_KV(pmix_client_globals.myserver, cb) — the data the server provided. For shmem3 this is the shared segments.

2. On a miss, DRAIN_KVS(cb), then — unless both ends are on hash, in which case the two peers point at the same tables — PMIX_GDS_FETCH_KV(pmix_globals.mypeer, cb), our own hash tables (where our own puts and any cached direct-modex answers live).

Either hit goes to process_values() and completes.

Note

What is deliberately not here is a retry at PMIX_RANK_WILDCARD. PMIX_RANK_UNDEF means “anything in this nspace”, and job-level data is part of that even though it is filed under no rank — answering it is the datastore’s job. Both in-tree modules once searched only the per-rank tables, and a client-side retry hid that for years. The retry is gone; test/unit/get_api.c holds the module behavior directly.

Going to the server. On a miss the caddy records cb->pname (the nspace/rank the reply will be matched against) and:

• a NULL key does not go up, unless the server is pre-v3.2 or we are asking about another namespace’s job-level info, in which case the request rank is rewritten to PMIX_RANK_WILDCARD;

• a reserved key goes up like any other — the server’s host frequently knows values it chose not to push down, and it is asked only if the request is surfaced to it;

• a server that is not a tool, a disconnected process, PMIX_ERR_EXISTS_OUTSIDE_SCOPE, or lg->optional all end the request here;

• otherwise the pending-request list is scanned with same_target() (exact rank and nspace). A matching outstanding request means this caddy is simply appended and no message is sent — the reply will satisfy both.

Finally _pack_get() builds a PMIX_GETNB_CMD message — nspace, rank, the (possibly rebuilt) info array, and the key if there is one — the caddy is appended to pmix_client_globals.pending_requests, and PMIX_PTL_SEND_RECV sends it with _getnb_cbfunc as the reply handler.

6.8.11.8. Step 7: the reply

_getnb_cbfunc() is a PTL receive callback, so it is already on the progress thread — no shift is needed. It unpacks the status, calls PMIX_GDS_ACCEPT_KVS_RESP to store the payload (for shmem3 this is where a client takes delivery of segment information rather than data), then walks every pending request matching the same target and answers each: fetch from mypeer, retry at PMIX_RANK_WILDCARD if the request was at PMIX_RANK_UNDEF, fall back to myserver’s module, then process_values().

An empty buffer means the connection was lost, and every waiter is told PMIX_ERR_LOST_CONNECTION rather than the “not found” the status was seeded with — those are different facts.

Completion for each waiter is either gcbfn() (non-blocking: invoke the caller’s callback, release lg and the caddy) or cb->cbfunc.valuefn — _value_cbfunc() for the blocking form, which sets cb->status/cb->value and calls PMIX_WAKEUP_THREAD(&cb->lock), releasing the caller in step 5.

6.8.11.9. Step 8: inside pmix_gds_shmem3_fetch()

This is the read of the shared segments, and it may be running on the application’s thread (step 5) or the progress thread (step 6).

a. Acquire the tracker. pmix_gds_shmem3_acquire_job_tracker() takes joblock, finds the tracker for the nspace, PMIX_RETAINs it, and drops the lock. No such tracker means PMIX_ERR_INVALID_NAMESPACE — for a reader, simply “nothing to read here”. The reference is released at the end, and it is the only synchronization the rest of the read has.

b. Is there anything to read? shmem3_fetch_from_job() asks the chain whether a job segment has been published. A tracker exists from PMIx_server_register_nspace time, long before register_job_info() builds a segment, so a fetch arriving in that window is PMIX_ERR_NOT_FOUND, not a fault. Whether there is modex data is the same question asked of modex_chain.

c. The whole-job case. NULL key with PMIX_RANK_WILDCARD means “a complete copy of the job-level info”: job data under PMIX_RANK_WILDCARD, then session info, then node info and app info across the job chain, then one PMIX_PROC_INFO_ARRAY per rank built from that rank’s entries.

d. Realm dispatch. The qualifiers are scanned for PMIX_SESSION_INFO / PMIX_NODE_INFO / PMIX_APP_INFO; failing that, the key is classified the same way process_request() did. Session requests go to fetch_sessioninfo() (which walks the session’s own chain newest-first). Node and app requests with a non-valid rank go to chain_fetch_nodeinfo() / chain_fetch_appinfo(), which consult every job segment, since an addition published later carries its own lists.

e. Choose the store. PMIX_INTERNAL, PMIX_LOCAL, PMIX_GLOBAL, PMIX_SCOPE_UNDEF and any wildcard-rank request read the job chain; PMIX_REMOTE reads the modex chain. Anything else is PMIX_ERR_BAD_PARAM.

f. The lookup (label ``doover``). For PMIX_RANK_UNDEF the fetch sweeps ranks 0..nprocs-1; a keyed request returns at the first hit. On the job pass only, it then also asks PMIX_RANK_WILDCARD, because job-level data is filed under no rank and PMIX_RANK_UNDEF means “anything in this nspace”. That extra lookup is confined to the job pass on purpose — the doover retry returns here for the modex, where job data never is, and repeating it would collect the same value twice, which the client’s process_values() would shape into a PMIX_DATA_ARRAY of duplicates.

For a specific rank it is a single job_fetch() or modex_fetch().

g. The scope retry. On success with PMIX_GLOBAL, and on failure with PMIX_GLOBAL or PMIX_SCOPE_UNDEF, useremote is set and control jumps back to doover to try the other store. Everything past that label dispatches on this flag rather than on a table pointer, because the modex is a chain of generations rather than one table — arriving there with the flag unset ran the job fetch against the modex data paired with the wrong key index, which missed on the ordinary path (an unqualified get arrives as PMIX_SCOPE_UNDEF and reaches the modex only through here).

h. The chain walks themselves.

• job_fetch() walks job->job_chain newest-first; per segment it calls pmix_hash_fetch() with that segment’s key index (an index is minted per segment, so table and index cannot be separated), then drop_tombstoned(..., UINT32_MAX, mark). A keyed request stops at the newest segment that has the key; a NULL-key request continues and drop_shadowed() removes anything a newer segment already supplied.

• modex_fetch() does the same over job->modex_chain, passing each segment’s own generation to drop_tombstoned(), and calls strip_undef() to remove entries whose value is PMIX_UNDEF — those are not data but the contributing process saying the key is gone, which hash acts on by removing the key and this component cannot.

i. Not found, but elsewhere. If nothing was collected and the rank was valid, a PMIX_LOCAL request checks the modex and a PMIX_REMOTE request checks the job data; if the key exists there, the result is discarded and PMIX_ERR_EXISTS_OUTSIDE_SCOPE is returned, so the caller is told rather than left to time out.

j. Final status. A non-empty list is success, whichever pass filled it. (modex_fetch() can report PMIX_ERR_NOT_FOUND after a successful NULL-key read of the current generation; this final check is what overrides it. It is fragile rather than wrong — if that check is ever made narrower, fix modex_fetch() first.)

6.8.11.10. Step 9: shaping the answer

process_values() turns cb->kvs into a single pmix_value_t:

• exactly one entry and a non-NULL key → that value, with ownership taken from the kval. A PMIX_QUALIFIED_VALUE is unwrapped to the value at element 0 of its embedded info array — after the shape is verified, because the datastore is also filled from the server and nothing screens it on the way out;

• an empty list → PMIX_ERR_NOT_FOUND;

• anything else → a PMIX_DATA_ARRAY of pmix_info_t, which is what a NULL-key get returns.

Back in PMIx_Get, lg->stval copies into the caller’s storage (and reports a failed transfer rather than handing back a type with nothing behind it), otherwise *val takes the value. A success with nothing behind it is converted to PMIX_ERR_NOT_FOUND: PMIx_Get entitles the caller to dereference *val when it says success.

6.8.11.11. Summary of thread shifts

Point	Mechanism	When it happens
PMIx_Get → get_data()	PMIX_THREADSHIFT, then PMIX_WAIT_THREAD	only when try_local_fetch() declines
PMIx_Get_nb → get_data()	PMIX_THREADSHIFT, returns immediately	same
PMIx_Get_nb answered by process_request()	PMIX_THREADSHIFT(cb, gcbfn)	PMIX_PROCID / PMIX_VERSION_NUMERIC / PMIX_RANK, so the callback still fires from the progress thread
refresh_cache()	no shift; blocking PMIX_WAIT_THREAD on a PTL round trip	PMIX_GET_REFRESH_CACHE
_getnb_cbfunc(), refcb()	none needed — PTL receive callbacks already run there	every server reply
try_local_fetch() → pmix_gds_shmem3_fetch()	none — runs on the caller’s thread	keyed get, no realm redirection, is_tsafe module

Note what PMIx_Get never does: it never takes a lock on the datastore it reads. The only lock in the whole path is the brief joblock around finding a tracker, and the only wait is the one a blocking caller does on its own completion.

6.8.12. MCA Parameters

Parameter	Default	Meaning
gds_shmem3_segment_size_multiplier	1.0	Scales the computed segment sizes. Raise it if a job overflows a segment (which aborts); prefer this over shaving the estimates.
gds_shmem3_arena_slot_size	1 GiB	Address space reserved per modex slot. 0 disables the arena and restores independent placement. Virtual address space only — nothing is committed.
gds_shmem3_arena_modex_slots	4	How many modex generations the arena can hold at once, capped at 32. More than one is live only where deltas are in play; past the last slot a generation is placed outside the arena.
gds_shmem3_arena_job_slots	8	How many job segments the arena reserves room for. The job’s own is the first; each addition a host makes to a registered job publishes another, since a published segment is never rewritten. A segment past the last slot is placed independently, and a client may then fail to map it where the server put it.
gds_shmem3_offset_placement	true	Place segments a quarter of the way into the biggest hole rather than at its midpoint.
gds_shmem3_force_client_attach_failure	false	Testing only — force every client attach to fail, so the init-time GDS fallback can be exercised.
gds_shmem3_force_modex_attach_failure	false	Testing only — force only the modex attach to fail. This is the case the parameter above cannot reach, since that one leaves the client on hash before it ever reaches a fence.
gds_shmem3_force_update_attach_failure	false	Testing only — force a job or session segment delivered after PMIx_Init to fail. Neither parameter above reaches that path. Drives test/unit/update_attach_fail.

pmix_hash_proc_alloc (src/runtime/pmix_params.c) is also relevant: it sets how many key slots each rank’s pointer array starts with, and for shmem3 that inflates the segment estimate for every rank in the job.

One more switch belongs to this component’s behavior without being an MCA parameter. Setting PMIX_GET_ON_PROGRESS_THREAD in the environment — to any value; only its presence is read — clears pmix_client_globals.fast_get, so try_local_fetch() declines every request and each PMIx_Get is thread-shifted the ordinary way. It is read once, in pmix_rte_init() (src/runtime/pmix_init.c).

It is deliberately not an MCA parameter, and so does not show up in pmix_info. An MCA parameter is a knob offered to a user, and no user has a reason to ask for a slower answer to the same question. This one is for a developer: it is how a datastore suspected of answering differently depending on which thread asked gets tested both ways, and how the test harness exercises the ordinary thread-shifted path on a build where shmem3 would otherwise short circuit past it. The dockerswarm suite runs datatypes and modex_twice both ways for exactly that reason.

6.8.13. Testing

shmem3 gets no coverage at all on macOS — configure.m4 gates it on a 64-bit, non-Apple host with no --enable-test-build escape, so it is not even compiled there.

• contrib/dockerswarm/run-gds-tests.sh is the suite that builds it on Linux and then exercises it: server and clients on shmem3, a client forced onto the fallback path, cross-node fetches that reach the modex, and examples/modex_twice.c run both cumulatively and with pmix_server_fence_delta_modex=1.

• test/unit/gds_datastore covers the framework contracts this component has to honor, against whichever module is assigned. Its last case, test_shmem3_job_segment(), asks for shmem3 by name and drives it end to end in one process — register an nspace, build the job and session segments, read a job-level key and the whole job back out, deregister — printing SKIP where the component is not available.

• test/unit/update_attach_fail and test/unit/session_update each fork and exec themselves as a real client behind a real socket, which is the only way to reach the delivery paths above from make check: the first asserts that a failed update attach costs only its own segment, the second that a session update reaches a running client.

• test/unit/get_api.c holds the PMIX_RANK_UNDEF behavior described in step 8f directly, from a real client.

• test/unit/util/util_vmem.c pins the placement and scatter properties against the placement function, where they are decidable — ASLR hides them in any live test.