PMIx v7.x series
================

This file contains all the NEWS updates for the PMIx v7.x
series, in reverse chronological order.

7.0.0 -- TBD
------------
Detailed changes since v6.1.0:
 - The pgpu/amd and pgpu/nvd components now activate on the nodes that
   have their vendor's GPUs. Both asked the topology for one exact PCI
   class/subclass, and one vendor's GPUs do not agree on a subclass: a
   card reports 0x0300 (VGA compatible), 0x0302 (3D controller) or 0x0380
   (other) depending on the part and on whether a display is wired to it.
   They now match the display base class 0x03 and ignore the subclass.
   pgpu/amd was additionally asking for PCI vendor id 0x1022, which is
   this vendor's id on its CPU and chipset functions rather than the
   0x1002 its GPUs carry, so it declined to open everywhere. Nothing said
   so in either case - a component that believes the hardware is absent
   is indistinguishable from one that is correct about it.
 - A process mapped against Intel GPUs is now told which ones it may use.
   The pgpu/intel component sets ZE_AFFINITY_MASK at fork time, naming
   each assigned card by the Level Zero device ordinal that node's own
   driver reported for it. The component previously set nothing, on the
   grounds that the variable takes ordinals rather than an identity of
   the kind CUDA_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES accept, and
   that PMIx could not reproduce the runtime's device ordering. It does
   not have to: hwloc's Level Zero backend records the ordering the
   driver itself reported, so the ordinals are read rather than guessed,
   and they are read on the node that will run the process.
   Because ZE_AFFINITY_MASK is interpreted against whichever devices
   ZE_FLAT_DEVICE_HIERARCHY exposes - whole cards under COMPOSITE,
   individual tiles under FLAT - an ordinal alone is not a complete
   statement, and PMIx now writes the model alongside it. A process that
   has already been given a model that contradicts the one this node
   enumerated under keeps every device it could see and is told, in
   place of a mask that would silently have handed it half the hardware
   it was assigned.
 - A deletion now reaches the data that has already been handed out.
   Removing a key from a PMIx server's own store was only half of it: a
   client caches what it reads about other processes and holds the
   job-level data it was given at initialization, so every local client
   that had looked the key up still had it. The server now tells its
   local clients, on a PTL tag of its own so that a peer too old to know
   about deletion never receives one. It goes to every local client
   except the one that asked, and is not restricted to the affected
   namespace, since a process may have cached data belonging to any
   namespace it asked about.
   This is also what answers PMIx_server_deregister_resources. The global
   cache is copied into a namespace's datastore once, when that namespace
   is first registered, and nothing re-reads it - so a deregistration
   governed only the namespaces registered afterwards and left every
   running job with its copy. It now takes the key back from the
   namespaces that already hold it, and from their clients. A qualified
   deregistration that prunes elements out of an entry rather than
   removing it is deliberately not propagated: the host asked for part of
   a value to go, so a deletion would take more than was asked. A
   namespace using gds/shmem3 also still keeps its copy, in its shared
   segment. See openpmix#4087.
 - PMIx_Put gained four scope values - PMIX_DEL_LOCAL, PMIX_DEL_REMOTE,
   PMIX_DEL_GLOBAL and PMIX_DEL_INTERNAL - naming the same audiences as
   their storing counterparts but directing that the key be removed
   rather than stored. The value is ignored and may be NULL, and removing
   a key that was never stored is not an error. The removal takes effect
   on the calling process at once and reaches the local server through
   the usual commit, so a PMIX_DEL_INTERNAL - which was never shared - is
   complete on return. This is a permitted extension: PMIx_Put(3) already
   states that an implementation may support additional scope values and
   must answer PMIX_ERR_NOT_SUPPORTED for one it does not, which is what
   a server predating these returns. That check is made up front, in
   PMIx_Put, because a server that does not recognize a scope would
   otherwise drop the block and let the delete appear to succeed.
   Companion projects can detect support through PMIX_CAP_DATA_DELETE.
   Propagating a removal to the other clients of a namespace, and to
   gds/shmem3's shared segments, is not yet implemented. See
   openpmix#4087.
 - A PMIx server now rejects a commit whose scope it does not recognize
   rather than silently discarding the data that scope labelled. It also
   rejects PMIX_INTERNAL, which names data that never leaves the process
   and so has no business on the wire.
 - A PMIx server can now contribute only what its processes published
   since they last took part in a collecting fence, rather than
   everything they have published, controlled by the new
   pmix_server_fence_delta_modex MCA parameter. It defaults to false: a
   server from a release that predates the delta marker rejects the whole
   collective rather than storing a contribution it cannot interpret -
   the right failure, but it means a job running mixed releases works
   today and would stop working if this defaulted on. Enable it once
   every node understands the marker. A delta is only sent when every
   local participant has already contributed to a fence over the same
   participant set; anything else falls back to the full set, which
   keeps two sub-communicators fencing independently from withholding
   data from each other. The watermark advances only once the host has
   taken the bucket, since the request has three arms that discard it.
 - gds/shmem3 keeps modex generations rather than always dropping the
   previous one. A cumulative contribution repeats everything, so it
   still supersedes the generation before it and every one behind that.
   A delta repeats nothing, so the previous generation is retired instead
   and a read walks the generations newest-first: a keyed lookup stops at
   the newest one holding the key, and a whole-process lookup consults
   all of them while dropping a copy of a key a newer generation already
   supplied. The chain is empty unless a delta has been stored, so the
   ordinary case is the single lookup it has always been, and any
   cumulative fence collapses it again. Segment blobs gained a field
   telling the client which kind it is looking at, so it makes the same
   decision its server made.
 - PMIx_Commit now transmits only what the process has posted since the
   previous commit. It used to fetch and send that process's entire local
   and remote store every time, so a single new PMIx_Put caused everything
   published so far to be sent again and n put/commit cycles moved O(n^2)
   bytes. PMIx_Put records which keys each scope owes the server and the
   commit fetches just those; a key posted repeatedly between two commits
   is still sent once, carrying the value it had when PMIx_Commit was
   called. The cumulative fetch remains as the fallback for the cases a
   per-key record cannot express - a PMIX_QUALIFIED_VALUE, which has no
   key a later fetch could ask for it back by; a tool that has repointed
   at another server through PMIx_tool_set_server or _attach_to_server,
   which has seen none of what we sent the previous one; and the first
   commit after PMIx_Init. Nothing changes for the caller, and nothing
   changes on the wire, so a client and server of different releases
   interoperate exactly as before. See openpmix#4087.
 - The per-server flag byte carried in the modex envelope is now screened
   before it is used. That byte says what kind of contribution a server
   made, and the only check on it was that the contributing servers agreed
   with each other - so a value they all agreed on, and that no datastore
   knew how to act on, passed straight through and its data was stored as
   though it were an ordinary full contribution. It is now rejected with
   PMIX_ERR_BAD_PARAM. A new value, PMIX_MODEX_DELTA, marks a contribution
   carrying only what the sending processes published since they last took
   part in a collecting fence, and the value is now handed to the
   datastore, which decides what it means for its own storage. Since the
   agreement check is what an older release already performs, a job mixing
   a release that sends delta data with one that cannot store it fails
   loudly on both sides instead of silently losing data. See openpmix#4087
   and the delta-exchange section of docs/how-things-work/modex.rst.
 - An MCA component whose framework interface version does not match the
   framework it is being loaded into is now refused instead of being
   opened. Nothing checked this before: the only version test in the
   loader compares the *MCA* major.minor, which is one number for the
   whole project and says nothing about whether a given framework's
   module struct has changed underneath the component, and its "TODO --
   add checks for project version (from framework)" has been there since
   the file was inherited from Open MPI (whose copy still has it, worked
   around by eight of its frameworks repeating the test themselves).
   Because components are run-time-loadable, an installed plugin older
   than the library loading it is what any partial upgrade produces, and
   it presents whatever module struct its header had at the time - which
   the library then calls through. pcompress grew two entry points in
   July 2026 without a version bump, and a plugin from before that left
   them NULL and turned the first size query into a jump to address
   zero. pcompress is accordingly bumped to 3.0.0, so such a plugin is
   now declined and the framework falls back to its own defaults - no
   compression rather than a crash. A refused component is reported
   through the usual mca_base_component_show_load_errors gate, which
   defaults to "none"; set it to the framework name (or "all") to see
   which plugins were dropped and why. Anyone carrying an out-of-tree
   pcompress component must rebuild it. A framework states its interface
   version as three macros in its own header, which both its components
   and its declaration read, so there is one place to edit when it
   changes; PMIX_MCA_BASE_FRAMEWORK_DECLARE is unchanged, and a framework
   declared with it - as every framework outside this project is - states
   no version and has its components checked as before
 - Every pcompress component now exposes its compression level as an MCA
   parameter - pcompress_zlib_level, pcompress_zlibng_level and
   pcompress_zstd_level - where the zlib components previously hard-coded
   level 9. That is the wrong end of the speed/size curve for what this
   framework compresses: a large collective payload, on a single progress
   thread, with a whole job waiting on the result. A broadcast pays the
   deflate once and the wire cost on every link of the tree, so the last
   couple of percent of ratio only repays itself on a very wide tree over
   a slow link. Measured through PMIx on a 25.6 MB aggregated modex,
   single-threaded, zlib level 1 gives a ratio of 0.649 at 103 MB/s
   against level 9's 0.638 at 54 MB/s. The defaults are now 1 for zlib
   and 2 for zlib-ng - they differ by one deliberately, because zlib-ng
   remaps its level 1 onto a quick-deflate strategy that is much faster
   and appreciably weaker, so zlib-ng level 2 is what zlib level 1
   produces. The two defaults match in behaviour rather than in digit
 - Added a zstd component to the pcompress framework, ranked above the
   zlib ones (zstd 90, zlibng 75, zlib 50), so a build that finds
   libzstd uses it for every compressed payload. The zlib component
   deflates at level 9 unconditionally, which is the wrong end of the
   speed/size curve for what this framework is used for - shrinking a
   large collective payload on a single progress thread while the whole
   job waits on it. Measured on a 25.6 MB aggregated modex,
   single-threaded, zstd's default level 3 reaches a ratio of 0.588 at
   427 MB/s where zlib level 9 takes 64 MB/s to reach only 0.600, and
   decompression - which every peer pays rather than just the
   originator - runs at 3.6 GB/s against 0.6. The compression level is
   an MCA parameter, pcompress_zstd_level. Note that a zstd blob is not
   DEFLATE, so unlike zlib and zlib-ng the components are NOT mutually
   readable: every node in a job must run the same one. The zstd
   component checks the frame magic and refuses a foreign blob rather
   than inflating garbage
 - Added PMIx_server_IOF_flow_control, by which a host environment can
   suspend and resume the processes that are feeding stdin to it. A
   host falling behind on stdin previously had no way to slow its
   producers: the library discarded the status of every push_stdin
   upcall and re-armed the read unconditionally, so a refusal anywhere
   would have dropped bytes rather than throttled anyone. The library
   now applies such a request both to any stdin it is reading itself
   and to every tool that has pushed stdin to it, and a tool that is
   itself a server relays it onward - so a chain of launchers carries
   it back to whoever holds the input stream. Nothing is buffered on
   behalf of a suspended stream and nothing is lost; the bytes stay in
   the producer's input stream, where the OS applies the back-pressure.
   A host can also suspend opportunistically by completing a push_stdin
   upcall with the new PMIX_ERR_IOF_XOFF status, which means "I have
   taken this data, now stop sending" and is deliberately not a
   failure. Detectable at build time as PMIX_CAP_IOF_FLOW_CONTROL
 - The library no longer re-arms its stdin read before the far end has
   acknowledged the chunk it just sent. Both the server-role path (the
   push_stdin upcall) and the tool-role path (the relay to its server)
   used to fire and forget, which is what made the stdin producer
   impossible to pace; the tool path additionally armed the read event
   twice per chunk. Stdin now flows at the rate the consumer accepts it
 - A tool whose stdin forwarding is suspended is no longer told its
   forwarding failed. The tool-side handler treated every non-success
   acknowledgement as terminal - deleting the read event and raising
   PMIX_ERR_IOF_FAILURE - so a "slow down" from the far end would have
   killed the stream instead of pausing it
 - Fixed a crash on SIGCONT in a tool or launcher that is collecting a
   terminal's stdin. The signal handler is registered with no callback
   data, and the handler it shares with the stdin restart path
   dereferenced that as a read event
 - A process now records the alias forms of its own node name no matter
   where that name came from. The library builds the alias list - which
   is what lets it recognize its own node under both the FQDN and the
   short form - only when it had to discover the hostname itself, so a
   host environment that supplied PMIX_HOSTNAME, which every resource
   manager does, left the list empty. Every other node's name was
   already being passed through the same normalization, so ours was the
   only one not getting it
 - The list of CPUs to which the internal progress thread is to be bound
   is now validated before it is used. An entry that is not a number, a
   negative one, one beyond the end of the CPU mask, or a backwards
   range is reported and skipped; if binding was requested as required
   and nothing usable remains, initialization fails. Previously the list
   went to strtoul unchecked, which reports zero for a token containing
   no digits - so a typo such as "cpu0" silently became "bind to CPU 0"
 - Added the missing help text for a malformed or unrecognized entry in
   the pmix_var_dump_color MCA parameter. Both messages were being
   requested by name and neither existed, so a user who mistyped that
   parameter got "I couldn't find that help reference" in place of the
   explanation
 - Under PMIX_EXTERNAL_PROGRESS, finalize now releases the event base
   and marks the library as no longer accepting work. Both were being
   skipped along with the progress thread that mode does not start, so
   the base outlived finalize while roughly a hundred API entry points
   went on believing a torn-down library was open for business
 - Several allocations are no longer leaked on each init/finalize cycle:
   the output channels opened for the client verbosity parameters, the
   two static IOF sinks in a server, the file and directory names given
   by PMIX_IOF_OUTPUT_TO_FILE / _TO_DIRECTORY, and the environment-
   variable harvest patterns held by the pnet, pgpu and pmdl components.
   Also, directive state such as the node ID and the external-progress
   and external-topology flags is now reset, so what one PMIx_Init was
   told no longer becomes the default for the next one
 - The datastore now fills in the per-proc location keys - hostname,
   nodeid, local rank and node rank - for every rank a host did not
   describe itself, and for every one of those keys a host left out. It
   works these out from the node and proc maps, and it rightly declines
   to overwrite anything the host stated in a PMIX_PROC_INFO_ARRAY,
   since what it computes is only an assumption. But that decision was
   being made once for the whole job: a single proc-info array anywhere
   in the registration suppressed the derivation of nodeid, local rank
   and node rank for every rank in it. So a host that described one
   process lost those three keys for all the others, and a host that
   described every process but named only some of the keys lost the
   rest for all of them - PMIx_Get answering PMIX_ERR_NOT_FOUND with
   nothing to fall back on. The choice is now made per rank and per
   key. Hostname, which was being derived outside the same test, now
   follows the rule in both directions: it is supplied when the host
   gave none, and left alone when the host gave one
 - A batch of server-side request-handling fixes from a review of
   src/server. Two of them are use-after-free or double-free rather
   than leaks: the IOF pull and deregister handlers released the
   caddy the host had just accepted ownership of, so the host's
   completion callback ran on freed memory; and the fabric-register
   handler released its query caddy twice on a malformed request.
   Related, the internal response path for a fabric request the
   pnet layer answered itself handed the completion function a
   server caddy where it expected a query caddy, and the fabric
   update path left that function unset entirely
 - A server no longer answers a PMIx_Get for a rank that is not one
   of its own by reading a peer id out of a list sentinel. The value
   is arbitrary, and when it happened to name a live client slot the
   request was treated as local and parked to wait for a commit that
   was never coming
 - A deferred direct-modex request that the server could not launch -
   no host support, or a host that rejected the up-call - is now
   fully discarded. Only the tracker's own reference was being
   dropped, and since every parked requester holds one of its own,
   the tracker survived unreachable: off the pending list where no
   later resolve could find it, and still holding a pointer to the
   caddy that was about to be freed
 - A registration for a default event handler (one made with no event
   codes) now checks the notification cache, so an event that arrived
   before the handler registered is still delivered to it
 - PMIX_TIMEOUT is now read into a variable of the width the accessor
   is told to write. The fence, connect and get handlers were passing
   the address of a time_t while asking for a 32-bit conversion, which
   fills the wrong half of the field on a big-endian host
 - Assorted leaks on server error paths: the group id and participant
   array of a malformed group request, the scratch lists of a job
   control request that was rejected part-way through the directive
   scan, the retained caddy behind a failed fabric request, the reply
   buffer of a collective or get whose status could not be packed, and
   the payload already assembled for a cross-namespace get that then
   found nothing. The fence and connect timeout handlers also no longer
   release a tracker that is still linked into the collectives list
 - The fence and connect handlers no longer drive their completion
   callback on the internal error path that has no tracker to give it -
   in connect's case it was being handed a caddy instead, and in both
   cases the caller was then answered and released a second time by
   the switchyard
 - A server asked to resolve the peers of every known namespace now
   reports that it found none, rather than reporting success with an
   empty list, and gives each namespace its own first-choice lookup
   rather than letting one namespace's fallback rank change the search
   for all the ones after it
 - The fence collective's duplicate-contributor check now works. The
   list it consults was never appended to, so it was always empty: a
   clone sharing a rank with its parent had its remote data packed into
   the modex bucket twice, and one tracking object was leaked per
   contributor per fence. The per-rank blob that same loop assembles is
   also released now, rather than being leaked once the pack has copied
   it
 - A server no longer answers a notification addressed to several
   processes by rebuilding its target list out of copies of the one
   process being removed from it. The purge that runs when a peer departs
   indexed the surviving array by the wrong loop variable
 - When a process others are waiting on departs, the direct-modex
   requests parked against it are now failed rather than dropped. The
   tracker was being released while each waiting request still held a
   reference on it, so it survived unreachable and the waiting clients
   were left expecting a reply nobody would send
 - A namespace update that carries a group context id now stores it
   against the namespace rather than against whatever process identity
   happened to be left on the stack, and reports a failure of that store
   instead of the result of the preceding value copy
 - PMIx_server_finalize now releases the system temporary directory it
   recorded at init. Besides the leak, a second PMIx_server_init found it
   still set and silently kept the previous cycle's value
 - A server told to make an optional connection to another server, and
   unable to make it, no longer composes a PMIX_SERVER_URI out of the
   connection that did not happen
 - Assorted server leaks on paths that "can do nothing": five host
   callbacks that returned without honoring the release function the host
   gave them, the scratch lists of PMIx_server_register_resources, the
   directive array a launcher-spawned server uses to attach to its parent,
   and the result caddies of PMIx_server_setup_application and
   PMIx_server_collect_inventory when the caller supplies no callback
 - PMIX_GROUP_ENDPT_DATA is now documented with the type it actually
   carries: a pmix_data_array_t* of pmix_info_t, led by the contributing
   process' PMIX_PROCID and the PMIX_DATA_SCOPE at which the remaining
   elements are to be stored. The header had described it as a
   pmix_byte_object_t, which was the shape of a group-construct data
   exchange the library no longer performs. That stale type reached the
   generated attribute dictionary as well, so it is what the
   attribute-support queries and pattrs reported for the attribute
 - Every blocking PMIx entry point now reports PMIX_ERR_WOULD_BLOCK when
   called from within the PMIx progress thread, rather than hanging
   there. Making such a call has always been disallowed - the work that
   would release the caller is what that thread was about to do - but
   only a minority of entry points said so, and the rest simply stopped:
   an event handler that called PMIx_Fence, PMIx_Connect or
   PMIx_Group_construct wedged the process with nothing reported. The
   screen now covers all of them, sixty-one call sites, and emits a
   diagnostic naming the call. Where the wait is conditional on a NULL
   cbfunc - the fence, notify and event-registration entry points - only
   that path is screened, since with a callback those are meant to be
   driven from a handler. The affected man pages carry a PROGRESS THREAD
   RESTRICTION section stating the rule and the return code
 - PMIx_server_deregister_resources now honors the qualifiers its man page
   describes. A request element carrying a data array narrows the removal:
   PMIX_NODEID/PMIX_HOSTNAME choose the entries it applies to, and any
   other member of the array chooses the elements to remove from within
   them - matching an element directly or matching one that contains it,
   which is how a PMIX_FABRIC_DEVICE is named by its PMIX_FABRIC_DEVICE_NAME
   or PMIX_DEVICE_ID. Removing the last element an entry described removes
   the entry as well. Previously every such request matched on the key
   alone, so the man page's own example - remove one node's fabric device -
   deleted every node entry the server held. Note that deregistration
   governs the namespaces registered after it: non-namespace information is
   copied into a namespace's data store when that namespace is registered,
   so a job already running retains it
 - Fixed a double completion of a collective tracker that corrupted the
   server's collectives list and could abort a later PMIx_server_finalize
   with an invalid free. A collective the host never sees - a strictly
   local fence is the ordinary case - is finished by driving the tracker's
   completion function, which thread-shifts: until its handler runs the
   tracker is still on the collectives list and still tests as complete,
   because only that handler drains the participant list. Anything that
   walked the list in that window - the lost-connection sweep, a queued
   collective timeout - saw a complete, unclaimed collective and drove its
   completion a second time, and two handlers then each unlinked and
   released the same tracker. The host handoff was already guarded by
   host_called; a local completion had no equivalent. A tracker whose
   completion has been driven is now marked, and every path that can reach
   one honors the mark - including the tracker lookup, which would
   otherwise join a new contributor to a tracker about to be freed,
   hanging that client
 - A host that answers a spawn or direct-modex up-call with
   PMIX_OPERATION_SUCCEEDED is now told that it cannot, and the request
   is failed to its requestor rather than reported as a success carrying
   nothing. Both operations report a result - the namespace of the job
   that was launched, or the data that was fetched - and the callback the
   library supplies is the only channel for it, so an atomic completion
   had nowhere to put the answer: a spawn came back to the application as
   PMIX_SUCCESS with an empty namespace, over both the local and the
   remote path, with nothing said. The library always supplies that
   callback, so a host completing the work immediately can simply invoke
   it - before returning, if it likes - and return PMIX_SUCCESS. The
   pmix_server_module_t man page now states this, and PMIx_Spawn_nb no
   longer returns PMIX_OPERATION_SUCCEEDED, which the Standard does not
   define for it
 - A mistyped PMIX_DATA_SCOPE qualifier on PMIx_Get is now rejected with
   PMIX_ERR_BAD_PARAM rather than taken at face value, on both the client
   and the server side. The scope selects which table the datastore
   searches, so reading it out of a union that holds something else
   answered the request confidently and wrongly - it could not crash,
   since a scope is only ever compared and never used as an index, which
   is why it outlived the qualifiers beside it. Every other typed
   qualifier in the same code already rejected a type mismatch. The
   server-side read is the more exposed of the two: that info array
   arrives off the wire, so its type tag is the requesting peer's word
