Known gaps and deferred work
============================

This page inventories issues that were identified during the deep
reviews of the PMIx source tree but were **deliberately not fixed at
the time they were found**.  Each entry records what the problem is,
why it was left alone, and what closing it would take.

Three kinds of entry appear here:

* **Open decisions** — a real defect whose fix requires a judgement
  that the review was not entitled to make on its own (a change to a
  published attribute, a released API's behavior, or the PMIx
  Standard).
* **Deferred work** — a real defect whose fix is larger, riskier, or
  more entangled than the change it was found alongside.
* **Coverage gaps** — code that is believed correct but that no test
  reaches, usually because reaching it needs fault injection or
  hardware the test environments do not have.

Entries that are *by design* are also listed, under their own heading,
so that they are not repeatedly "rediscovered" and re-fixed.

The page opens with a fourth thing that is not an issue at all —
**Review coverage**, which records where the deep review has and has not
yet been, so that the remaining work is visible rather than rediscovered
each time somebody asks.

.. note:: Each directory's ``AGENTS.md`` carries the full reasoning for
          the items found in it.  Those files are orientation maps, not
          work logs, so this page is the place that records an item as
          still **open**.  Re-verify an entry before acting on it; the
          code moves.

.. note:: **Every entry below was checked against the tree on
          2026-08-11.**  That pass retired three of them — one whose fix
          had landed a month before this page first recorded it as open,
          one that had been fixed since, and one that a fresh valgrind
          run against a launcher could no longer reproduce — and
          corrected a claim that was never true.  Treat that as the
          standing expectation rather than a one-off: an entry here is a
          lead, not a finding.  The two "never validated" coverage gaps
          are the only remaining entries that cannot be checked by
          reading the tree, so they are where the next stale one will be.

          Three of the "smaller items carried forward" were closed on
          2026-08-13 — the ``PMIx_Value_get_number`` screen (now in the
          function itself, where every caller in the tree gets it), the
          ``init_called`` clear (now ``pmix_atomic_clear``, the required
          partner of the ``__atomic_test_and_set`` that sets it), and
          ``refcb()``'s dropped store status.

          **Checked again on 2026-08-15**, which retired one more: the
          open decision about a tool caching an event nobody handled
          while a client discarded it.  The fan-out fix for
          `openpmix#4101 <https://github.com/openpmix/openpmix/issues/4101>`_
          answered it — a client now parks such an event, the tool's
          unreachable copy of the parking code is gone, and
          ``test/unit/event_forward`` covers both halves.  Two entries
          were corrected rather than retired, both because the code had
          moved: the tool's SIGCONT stdin handler now lives in
          ``src/common``, and the ``cd->nondefault`` blocks are down to
          one that still assigns inside its guard.  Everything else was
          re-verified against the tree and stands as written.

          A second entry went the same way on 2026-08-16: the open
          decision about two concurrent spawns not being formattable
          differently.  It listed three unattractive choices and missed
          a fourth — hold the output until the reply that can identify
          it arrives, rather than formatting it on a guess at all.  The
          entry is retired; see ``test/unit/iof_pending``.  Worth noting
          *why* it read as closed, since the shape recurs: it framed the
          problem as "which flags does this output get", which really
          has no answer at that moment, when the question that does have
          one is "does this output have to be formatted yet".

.. note:: **2026-08-18.**  The ``psensor`` progress-thread entry that
          stood under "Deferred work" is closed.  Both halves it called
          for were made together: ``pmix_psensor_base_open`` now starts
          the ``"PSENSOR"`` thread it creates, and
          ``pmix_psensor_base_close`` pauses that thread before anything
          is torn down and stops it only after the components -- which
          own the trackers, and therefore the timers armed on the base --
          have closed.  Both components' samplers were moved to
          ``EV_PERSIST`` timers they no longer re-arm, for the reason
          spelled out in ``src/mca/pstat/AGENTS.md``.
          ``test/unit/run_monitor.pl`` now runs its heartbeat *and* file
          scenario twice, once with
          ``psensor_base_use_separate_thread`` set, so the configuration
          that was silently dead is exercised by ``make check``.

          A second entry was opened, corrected and closed the same day,
          and the shape of the mistake is worth keeping.  The
          unexpected-message ``show_help`` that ``run_monitor.pl``
          printed on every run was first written down as a heartbeat
          arriving at a server with no posted recv -- plausible, since
          ``psensor/heartbeat`` really does post that recv lazily, and
          wrong.  Tracing the tag showed the opposite: the server *does*
          match the beat, on the wildcard recv the command switchyard
          serves, and answers it with the error the switchyard gets for
          trying to read a command out of a zero-byte buffer -- and it is
          the *client* that then cannot place the reply, because
          ``PMIx_Heartbeat`` is one-way and nobody posts for tag 1.  Both
          halves are now fixed: ``pmix_server_message_handler`` drops
          ``PMIX_PTL_TAG_HEARTBEAT`` instead of dispatching it, and an
          unmatched message is a framework trace rather than a
          ``show_help`` asking the user to report a bug.  The lesson is
          the one this page keeps relearning -- an entry written from a
          plausible reading of the code, rather than from watching it
          run, is a lead and not a finding.

.. note:: **2026-08-20.**  The ``src/mca/psensor`` review found one
          shape repeated five times, and it is worth naming because it
          passes every test anyone would think to write: a directive
          parsed out of the request, stored in the tracker, and then
          never read again.  ``PMIX_MONITOR_HEARTBEAT_DROPS`` set
          ``ndrops`` and nothing consulted it, so a request asking to
          tolerate missed beats alerted on the very first empty window.
          ``PMIX_MONITOR_ID`` was never parsed at all, so the ``id``
          argument to ``stop()`` — the whole cancel handle — could never
          match anything.  ``PMIX_MONITOR_CANCEL`` was recognized by
          neither component, so it fell through to the resource-usage
          path, which answers ``PMIX_SUCCESS`` for an id it has never
          held: a client's cancel reported success and the monitor kept
          firing forever.  The ``error`` argument, which
          ``PMIx_Process_monitor(3)`` documents as "the code the monitor
          is to use", was discarded in favour of a hardcoded alert.  And
          the ``file`` sampler checked only the first of the three
          attributes a request named.

          Each of those looked like working code, and the framework's
          own end-to-end test passed throughout — because a test that
          waits for an alert cannot tell an honored directive from an
          ignored one.  ``test/unit/run_monitor.pl`` now arms a capped
          monitor and a cancelled one under a status code of their own
          and fails if *either* fires, which is the only shape that
          distinguishes them.  Each of the three fixes was verified by
          re-breaking it and watching the new check catch it.

          Two lifetime defects came out of the same pass.
          ``PMIx_Notify_event`` returns ``PMIX_ERR_NOT_AVAILABLE``
          without reaching its callback once the progress thread has
          stopped — which ``PMIx_server_finalize`` does well before it
          closes this framework — so both samplers stranded a tracker
          and the peer it retained.  And ``psensor/heartbeat`` never
          un-posted the PTL recv it posts lazily, nor cleared the flag
          saying it had: ``ptl`` closes after ``psensor``, so a DSO build
          left the ptl list naming a function in an unloaded plugin, and
          a second ``PMIx_server_init`` in one process would have found
          the flag set, the recv gone, and declared every monitored
          client dead on its first window.

.. note:: **2026-08-21.**  The ``src/mca/gds/shmem3`` re-review — the
          entry that stood under "Reviewed, but changed materially
          since" — is done, and three of its findings generalize past
          the component.

          **``is_tsafe`` is a claim about the reader's thread, not about
          the data.**  ``shmem3`` sets it, and ``try_local_fetch()`` in
          ``src/client/pmix_client_get.c`` consults it on every keyed
          ``PMIx_Get`` with ``pmix_client_globals.fast_get`` defaulting
          **on** — so the module's ``fetch`` runs on the application's
          own thread.  The justification recorded in the guide was that a
          fetch holds a reference on the job tracker and reads only data
          that is never written again.  That is true of what is *in* a
          segment, and of the job segment, and says nothing about the
          process-local bookkeeping a read walks beside it: the modex
          generation chain, which each completing fence releases and
          ``munmap``\ s, and the tombstone list, which ``del_key()``
          appends to.  A ``PMIx_Fence_nb`` concurrent with a
          ``PMIx_Get`` of a remote rank's key could leave an application
          thread inside ``pmix_hash_fetch()`` on a table that had just
          been unmapped; the single-threaded case was safe only by
          accident, because a blocking fence parks the app thread.  Fixed
          with a per-job mutex.  When reviewing any ``is_tsafe`` module
          the question to ask is not "is the data immutable" but "what
          process-local state does the read touch, and who else writes
          it".

          **A doc comment saying a field is "X-side only" has to be
          checked against every** *reader*, **not just the writer.**
          ``job->modex_generation`` was introduced to name the next
          backing file, which is a server-side job, and was documented
          and treated as server-only.  It is also what dates a tombstone,
          and ``del_key()`` runs on the client — where the counter never
          advanced, so every tombstone was stamped generation zero and
          shadowed every generation that client would ever map: a key
          deleted and re-published in a later fence stayed invisible to
          it.

          **A modex is stored through the module of the namespace that
          contributed it, not through the server's own.**  A server
          assigns *itself* ``hash``, while ``PMIX_GDS_STORE_MODEX``
          resolves from a local peer of the contributing namespace — so a
          ``shmem3`` job's fence data went into a shared segment that the
          lookup at the top of ``pmix_server_get()`` never searched.
          Every remote get for such a job missed and was pushed up to the
          host as a direct modex for data the server already held; if the
          owning process had finalized, nothing answered and the
          requester waited forever, because a remote request carries no
          timeout by design.  ``_satisfy_request()`` a few hundred lines
          below already had the right idiom — grep for
          ``local_peer_of_nspace`` before writing a second one.  Both
          halves were closed: the server now asks the namespace's own
          module, and ``_dmodex_req()`` answers ``PMIX_ERR_NOT_FOUND``
          for a rank the host has already reaped rather than deferring
          the request forever.

          One method note, because it decided the diagnosis: **a healthy
          run's duration is the yardstick**.  The case that exposed this
          takes about a second against a twenty-second limit, so a
          timeout there is a wedge and not a slow launch.  Measure the
          healthy case before calling a timeout flaky.

.. note:: **2026-08-24.**  The per-file pass over ``src/server``
          finished — twenty files, one at a time, after the
          directory-wide review of 2026-08-09/15 — and most of what it
          could not close itself landed here rather than in a commit:
          the entry-point initialization sweep below, the ownership of a
          credential the host hands up, the cleanup request that is
          applied in part before it fails, the server-wide envar hook
          nothing fills, and the allocation-failure injection the
          switchyard's out-of-memory arms need before any of them can be
          tested.  One entry was opened and closed within the same day —
          the ordering of a registration's acknowledgement against the
          cached events replayed behind it — because the push-back that
          had deferred it ("libevent ordering is too strong an
          assumption") was checkable and wrong: a single progress thread
          draining activations in order is a design invariant, not an
          assumption.

          That is the lesson worth carrying, and it is this page's own
          rule pointed at a different artifact.  **A recorded push-back
          is a lead, not a finding**, exactly as an entry here is.  The
          switchyard's unchecked ``PMIX_NEW`` in ``PMIX_GDS_CADDY`` and
          ``PMIX_SERVER_QUEUE_REPLY`` had been logged in ``AGENTS.md``
          as deliberate, on the reasoning that a NULL-safe macro "only
          moves the crash to the handler on the next line".  The dispatch
          arm returns before the handler is called, so the reasoning was
          simply false, and one failed allocation was killing the server
          and every client it hosted.  When a push-back's stated reason
          can be checked, check it before inheriting it.

Review coverage
---------------

Assessed on **2026-08-15** from the commit history and refreshed as each
review lands: on **2026-08-20** for the ``src/mca/pnet``, ``src/mca/preg``,
``src/mca/pgpu``, ``src/mca/pmdl``, ``src/mca/pcompress``,
``src/mca/plog``, ``src/mca/psensor``, ``src/mca/psec`` and
``src/mca/pif`` reviews, on **2026-08-21** when ``src/mca/gds/shmem3``
was re-reviewed against its redesign, and on **2026-08-24**, when the
per-file pass over ``src/server`` finished — all twenty ``.c`` files —
and the last of ``src/client`` came inside a review.  Move an entry out
of "Not yet reviewed" as its review lands, and refresh the churn figures
in "Reviewed, but changed materially since" when a re-review closes one.

.. note:: **An** ``AGENTS.md`` **is not evidence of a review.**  Every
          directory under ``src/`` has one; most were written in the July
          2026 orientation sweep, before any review started.  What marks
          a reviewed directory is a run of repair commits — "Repair…",
          "Screen…", "Close the … holes", "Sweep the leftovers in…" —
          and a commit that records the result in the guide.

Reviewed and current
^^^^^^^^^^^^^^^^^^^^

``src/class``, ``src/client``, ``src/event``, ``src/include``,
``src/runtime``, ``src/server``, ``src/tool``, ``src/tools``,
``src/mca/base``, ``src/mca/bfrops``, ``src/mca/gds/base``,
``src/mca/gds/hash``, ``src/mca/gds/shmem3``, ``src/mca/pcompress``,
``src/mca/pgpu``, ``src/mca/pif``, ``src/mca/plog``, ``src/mca/pmdl``,
``src/mca/pnet``, ``src/mca/preg``, ``src/mca/psec``, ``src/mca/psensor``,
``src/mca/pstat``, ``src/mca/ptl``, ``src/threads``, and
``bindings/python``.
``src/common``, ``src/hwloc`` and ``src/util`` were reviewed too, but
have moved since — see below.

**``src/client`` and ``src/server`` are the only two directories the
review has taken file by file**, a dedicated pass per file rather than a
directory-wide one, and the difference is worth stating rather than
flattening: a directory-wide sweep is real coverage, and it is not the
same thing.  The dedicated pass found something in *every* file it
read, including files that had already been through four or five
directory-wide sweeps.  ``src/client`` reached that state on 2026-08-23
and ``src/server`` on 2026-08-24 — eleven files and twenty, five lenses
each, one file to a pass.  The per-file record is in the 2026-08-23 and
later rows of ``.git/deep-review/ledger.tsv``; the reasoning is in each
directory's ``AGENTS.md``.

Two files fall short of that and are named here rather than rounded up,
both in ``src/client``:

* ``pmix_client_fence.c`` has had **no dedicated pass**.  It was signed
  off by the *directory-wide* five-lens seventh sweep — recorded there as
  coming through all five lenses with nothing to fix — and has not moved
  since.
* ``pmix_client_get.c``'s dedicated pass ran in two parts, and only the
  second one finished it.  The first (2026-08-22) covered lens 1
  (memory) over ``process_request``, ``PMIx_Get``, ``PMIx_Get_nb``,
  ``gcbfn`` and ``try_local_fetch`` and stopped there; ``get_data``
  (~500 lines), ``_getnb_cbfunc``, ``process_values`` and
  ``refresh_cache``/``refcb`` went unread, and lenses 2–5 were never
  applied to any of it.  The work that followed the same day — the realm
  and job-level-data fixes, the NULL-key answer, the ownership contract
  now stated in ``PMIx_Get(3)`` — chased specific findings out of that
  partial pass rather than completing it.  **All five lenses have since
  been applied to the whole file (2026-08-25)**, which is what found the
  entry below.

  What that second part found is worth stating here, because it is a
  property of the *list* rather than of either function that walks it.
  ``pmix_client_globals.pending_requests`` is two tables in one — the
  coalescing table ``get_data()`` consults before it sends, and the
  delivery table ``_getnb_cbfunc()`` walks when a reply arrives — and
  they matched on different predicates: ``PMIX_CHECK_NAMES``, which
  treats ``PMIX_RANK_WILDCARD`` on either side as a match, against exact
  rank equality.  A get for a specific rank issued while a get at
  ``WILDCARD`` for the same namespace was outstanding was therefore
  folded onto it, never sent, and never matched by the reply — and
  nothing else drains that list, so a blocking ``PMIx_Get`` waited
  forever and a ``PMIx_Get_nb`` was never called back at all.  Both
  sites now use one exact predicate; ``test/unit/get_api.c`` covers it.
  The general shape to look for: **one list serving two questions is
  one question, and a request the first accepts and the second rejects
  is not refused, it is lost.**

Not yet reviewed
^^^^^^^^^^^^^^^^

Each of these has an orientation guide and nothing else: no findings were
ever recorded in it, and only drive-by fixes have landed.  Ordered by
size, which is a rough proxy for how much there is to find.

* ``src/util/keyval`` — 2397 lines of lexer and parser, and the only
  directory in ``src/`` with **no** ``AGENTS.md`` at all.
* ``src/mca/pdl``, ``src/mca/pinstalldirs`` — about 2200 lines between
  them, and the lowest risk of the group.

Outside ``src/``, nothing has been reviewed: ``examples/`` (16678 lines,
leak-swept only), ``test/simple`` (11011), ``test/unit/util`` and
``test/unit/mca``, and the public headers in ``include/``.

Reviewed, but changed materially since
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ordered by how much of the directory the review no longer covers.
Figures are against the commit that recorded each review, measured
2026-08-24.

#. **``src/util``** — reviewed 2026-07-17/18, the oldest review in the
   tree, and the gap keeps widening: 38 commits and +2024/-247 across 21
   files since, none of it re-read.  The largest pieces are the CLI
   option-parsing rework, the ``dirpath`` conversion to descriptor-based
   operations, and the ``pmix_hash`` qualifier arrays.
#. **``src/hwloc``** — reviewed 2026-08-02.  +1129/-205 across three
   files since — the device enumerator and the reworked distance
   computation, against a five-line touch to the guide.  Effectively new,
   unreviewed code.
#. **``src/common``** — reviewed 2026-08-02, and no longer the modest
   churn it was listed as.  ``pmix_iof.c`` has 1215 changed lines since:
   the tool being given the library's stdin forwarding instead of its
   own, the hold on a spawned job's early output, and the SIGCONT
   handler that moved here out of ``src/tool``.  ``pmix_pfexec.c`` has
   335, most recently the removal of the library's signal traps and the
   per-holder reference on a pfexec child.  The other fifteen files are
   close to what the review read, so a re-review is really a re-review of
   those two.

Lower priority, with current guides and churn that is mostly the reviews'
own work coming back through: ``src/event`` (+414/-29 since 2026-08-01,
nearly all of it the notification fan-out and cached-event replay that
the ``src/server`` and event-forwarding reviews drove), ``src/mca/ptl``
(+617/-365 since 2026-08-07, the connection handler plus deletions
elsewhere), ``src/mca/gds/hash`` (+495/-152 since 2026-08-03, chiefly the
single job-level store the ``PMIx_Get`` review forced), and
``src/mca/base`` (+17/-2 since 2026-08-16).

Open decisions
--------------

A conflicting cleanup request is applied in part before it fails
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``pmix_server_job_ctrl`` (``src/server/pmix_server_control.c``) applies a
request's ``PMIX_CLEANUP_IGNORE`` directives to the target epilogs first,
then its ``PMIX_REGISTER_CLEANUP_DIR`` and ``PMIX_REGISTER_CLEANUP``
directives — and the latter two fail the whole request with
``PMIX_ERR_CONFLICTING_CLEANUP_DIRECTIVES`` when a path they were asked to
clean is already on that epilog's ignore list.  By then this request's
ignores, and any directories accepted ahead of the conflicting one, are
already registered on lists that outlive the request.  So the client is
told the request failed while the server keeps part of it.

What survives is the conservative half — a path ends up ignored rather
than deleted — which is why this is recorded rather than repaired.
Making the request atomic means splitting the walk into a validation pass
and an application pass, and the validation pass has to reproduce the
duplicate-detection logic exactly or it will reject requests the current
code accepts (a directory that duplicates an already-registered entry
takes the flag-upgrade branch today and is never conflict-checked at
all).  Whether a partially conflicting job-control request should be
atomic is a question for the Standard rather than for this file.

The same shape applies to the allocation-failure arms added alongside
it: a ``strdup`` that fails midway through the walk returns
``PMIX_ERR_NOMEM`` with the earlier entries registered.  That one is not
worth a rollback on its own, but it would come out in the wash of a
validate-then-apply split.

Who owns a credential the host hands up
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``pmix_credential_cbfunc_t`` is documented in ``include/pmix_common.h.in``
as transferring ownership of the credential to the receiving function —
"responsibility for releasing the memory lies outside the PMIx library."
That text is written for the *client* side, where the receiving function
is the application's callback.  Read literally it also governs the
server-side up-call, where the receiving function is
``pmix_server_cred_cbfunc``: the host's ``pmix_byte_object_t`` would then
be ours to free.  It does not free it — it makes a copy and leaves the
original alone — so under that reading every ``PMIx_Get_credential``
leaks the host's credential, and under the opposite reading freeing it
would be a double free in the host.

Deciding this needs the Standard's word rather than this file's, and the
wrong guess is much more expensive in one direction than the other, so
the copy stays.  If the transfer reading is confirmed, the fix is to take
the host's object rather than copy it (``psec`` mechanisms allocate a
fresh one per request), not to add a ``free`` beside the copy.  The same
question applies to the info array on the same callback, which the same
paragraph describes as owned by the PMIx library.

A pruned deregistration is not propagated
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This entry used to say that
:ref:`PMIx_server_deregister_resources(3) <man3-PMIx_server_deregister_resources>`
could not take information back from a namespace that already held it —
the server's global cache is copied into a namespace's data store once,
when that namespace is first registered, and nothing re-reads it — so a
deregistration governed only the namespaces registered *after* it while a
running job kept its copy.  **That is closed.**  ``PMIx_Put`` gained
delete scopes, ``gds`` gained a ``del_key`` slot, a deregistration takes
the key back from every namespace that already holds it, and the server
tells its local clients so their cached copies go too.

Two things about how it closed are worth keeping, because both contradict
what this entry predicted.

``gds/shmem3`` did **not** need a new segment generation.  The obstacle
was always that a client reads the shared segment directly, so removing
data from one would mean putting a lock on a read path that is lock-free
by design — and the assumed answer was to publish a new generation
carrying a ``PMIX_UNDEF`` tombstone, which there is no way to advertise
outside a fence reply.  The record does not have to be *in* the segment.
Each process keeps its own list on ``job->tombstones``, built from the
notification its server sends, and every read consults it;
``pack_tombstones()`` adds the list to the cached job-info reply so a
client attaching later gets it too.

And the deletion had to be made to cross a node boundary, which is a
separate problem from the one this entry describes.  ``deregister_resources``
does not need it — the host calls that API on every daemon — but a
``PMIx_Put`` deletion happens in one process, and the modex is additive,
so a contribution that merely stops naming a key removes nothing at the
far end.  A removal is therefore *stated* in the next collecting fence, as
an entry whose value is ``PMIX_UNDEF``, and both datastores act on one
that arrives.  ``examples/delete_key.c`` is the case that proves it, and
it only proves anything with one rank per node.

What remains is the *qualified* form, and it is deliberate rather than
missing.  A deregistration that prunes elements out of an entry instead of
removing it is not propagated: the host asked for part of a value to go,
so the right answer is the pruned value rather than a deletion, and that
needs an update push which does not exist.  Building one means deciding
what a partial update looks like on the wire and in a shared segment
neither datastore can rewrite in place.

See :ref:`Delta Exchange and Data Deletion <modex-delta>` in
:doc:`how-things-work/modex`, tracked as `openpmix#4087
<https://github.com/openpmix/openpmix/issues/4087>`_.

A fabric plane claimed asynchronously is not recorded
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/pnet`` review (2026-08-19).
``pmix_pnet_base_register_fabric`` records a tracker for a plane only when
a module answers ``PMIX_OPERATION_SUCCEEDED`` — the "already done, inline"
status.  A module that claims the plane the *asynchronous* way, returning
plain ``PMIX_SUCCESS`` and calling back later, is never recorded, yet both
callers — ``src/server/pmix_server_fabric.c`` and
``PMIx_Fabric_register_nb`` — read ``PMIX_SUCCESS`` as "claimed, wait for
the callback".  A later update or deregister on that plane then answers
``PMIX_ERR_BAD_PARAM``, because the scan finds nothing.

Which half is wrong is the decision, and it cannot be settled by reading
the tree: **no shipped component implements** ``register_fabric`` **at
all** — grep the four component directories and the slots are unset — so
there is nothing to test a change against, and choosing the async
contract now pins down the interface a real fabric component would have
to meet.  Two loose ends belong to the same decision.  ``fabric->module``
is deliberately never set, so the branches in ``update_fabric`` and
``deregister_fabric`` that cast it back to a ``pmix_pnet_fabric_t *`` are
dead and every lookup goes through the index/name scan — a tracker
pointer parked in a caller's object would dangle the moment the fabric
was deregistered.  And nothing frees ``pmix_pnet_fabric_t.payload``:
``ftdes`` releases only ``name``, and the base never tells a component
its tracker died, so a component that parks an allocation there must free
it from its own ``deregister_fabric``.

See "The fabric path is scaffolding" in ``src/mca/pnet/AGENTS.md``.

Who owns an ``mca_base_*`` parameter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/pmdl`` review (2026-08-19).  ``pmdl`` re-prefixes
every value it reads out of an MCA param file so it reaches the library
that will look for it — ``PMIX_MCA_``, ``PRTE_MCA_`` or ``OMPI_MCA_`` —
and decides which by the parameter's first segment.  Two segments name
something in more than one library: ``mca`` (all three have an MCA base)
and ``pmix`` (OPAL carried a ``pmix`` framework of its own).

Half of that overlap is now settled, because one side of it was plainly
wrong: ``parse_file_envars`` was claiming those names out of the list
PMIx read from **its own** param files, so a value the user set for PMIx
— ``pmix_hwloc_topo_file`` is the concrete one — was renamed
``OMPI_MCA_*`` and reached neither library.  A parameter PMIx claims is
now left alone there.

The other half is left as it stands.  ``process_param_file`` reads
``openmpi-mca-params.conf``, tests for a PMIx parameter first, and so
forwards ``mca_base_component_path`` from **Open MPI's** file as
``PMIX_MCA_mca_base_component_path``.  Note which name that is: PMIx
registers the variable as project ``pmix``, framework ``mca``, component
``base``, and the *full* name a param file and an envar carry leaves the
project off — so ``mca_base_component_path`` is PMIx's own spelling of
it, and ``pmix_mca_base_component_path`` is the project-qualified long
name, which a param file also accepts (see ``var_set_from_file``, which
matches either).  Open MPI spells its equivalent the same way, which is
the whole difficulty: the unqualified name is genuinely ambiguous, and
nothing in the line says which library it was meant for.  Claiming it for
PMIx has been the behavior for releases and a site may be relying on it.
Deciding it means deciding whether the two ``mca_base`` namespaces are
one setting or two.

The ``ompi`` component's guide records the precedence as it stands.

Deferred work
-------------

Smaller items carried forward
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The items that stood under this heading were closed on 2026-08-13; what
is left of them is recorded here, because in two cases the fix does not
cover the whole of what the entry described.

* **The pre-v3.2 branch of** ``resolve_peers()`` **no longer carries a
  dead store**, but what it *meant* to do is still not done.  The branch
  assigned ``PMIX_RANK_WILDCARD`` and was then overwritten with
  ``PMIX_RANK_UNDEF`` before anything read it; the assignment is gone and
  the branch now varies only the ``key`` and ``ninfo``, which is what it
  always really did.  The legacy path resolves because ``try_fetch()``
  retries an ``UNDEF`` rank as ``WILDCARD``.  **Fetching at ``WILDCARD``
  directly, as the branch intended, is still untried** — that is a
  behavior change on a path only a pre-v3.2 server exercises, and there
  is none to test against.
* **A malformed** ``PMIX_QUALIFIED_VALUE`` **or a NULL key arriving in a
  cache refresh now fails the enclosing** ``PMIx_Get``.  See the
  ``refcb()`` entry in ``src/client/AGENTS.md`` for why all-or-nothing is
  the only answer an application can act on.  Recorded here because it is
  the one behavior change in that group of fixes.

Most ``PMIx_server_*`` entry points screen the wrong initialization flag
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found reviewing ``src/server/pmix_server_inventory.c`` (2026-08-24).

Twenty-seven sites in ``src/server`` open with

.. code-block:: c

   if (!pmix_atomic_check_bool(&pmix_globals.initialized)) {
       return PMIX_ERR_INIT;
   }

and every one of the man pages behind them describes that return as "the
PMIx **server** library has not been initialized".  The flag does not
answer that question: ``PMIx_Init`` and ``PMIx_tool_init`` set it too.
And until ``PMIx_server_init`` has run, every list and pointer array in
``pmix_server_globals`` is only ``PMIX_LIST_STATIC_INIT`` /
``PMIX_POINTER_ARRAY_STATIC_INIT``, and the ``pnet``, ``pgpu``, ``pmdl``
and ``psensor`` frameworks these entry points fan out to have never been
opened.  ``PMIX_LIST_STATIC_INIT`` leaves the sentinel's ``next`` pointer
NULL, so a ``PMIX_LIST_FOREACH`` over one of those lists dereferences
NULL rather than finding it empty.  A tool or client process that calls
one of these APIs is therefore answered ``PMIX_SUCCESS`` and then takes a
SIGSEGV, usually on the progress thread after the call has returned.

``pmix_server_globals.initialized`` is the flag that does answer it — set
at the foot of ``PMIx_server_init``, cleared at the top of
``server_teardown()``.  ``PMIx_server_collect_inventory`` and
``PMIx_server_deliver_inventory`` now use it, and
``test/unit/server_inventory.c`` holds them to it (the refusal cases run
in a forked child, because against an unfixed library the crash lands
after the call has already returned).
``PMIx_server_define_process_set`` and ``_delete_process_set`` were
closed the same way (2026-08-24), covered by the forked cases in
``test/unit/progress_threads.c``; a client reaching either one SIGSEGVs
on the progress thread against an unfixed library.  The four
registration entry points followed (2026-08-24) — ``_register_nspace``
and ``_register_client`` walk ``pmix_server_globals.collectives`` and
``_deregister_nspace`` fans out to ``pnet``, ``pgpu`` and ``pmdl`` —
covered by the forked cases in ``test/unit/server_registration.c``.
``PMIx_server_register_resources`` and ``_deregister_resources``
(``pmix_server_setup.c``) followed (2026-08-24): both append to and walk
``pmix_server_globals.gdata``, so a client took SIGSEGV on the progress
thread after being answered ``PMIX_SUCCESS``.  Covered by the forked
cases in ``test/unit/server_setup.c``.

**``PMIx_server_setup_application`` and ``_setup_local_support`` were
tried and reverted, and the reason is the general warning for the rest of
this sweep.**  They were screened in the same change and it broke every
PRRTE ``prun`` launch with ``PMIX_ERR_INIT``.  A launcher comes up
through ``PMIx_tool_init``, which opens ``pmdl`` unconditionally and
``pnet`` for a launcher precisely so it can ask a server to launch
something, and ``prun_common.c`` calls ``PMIx_server_setup_application``
in that state without ever calling ``PMIx_server_init``.  The frameworks
already answer per role one level down —
``pmix_pmdl_base_harvest_envars`` returns ``PMIX_ERR_INIT`` when ``pmdl``
was never opened — so the entry-point screen added nothing and refused a
legitimate caller.  **Before screening any remaining entry point, check
its callers in PRRTE and not only in this tree**, and check specifically
for a launcher reaching it after ``PMIx_tool_init``.

What is deferred is the sweep over the rest, because the answer is not
uniform and each entry point needs checking against its real callers
first:

* **Almost certainly want the screen** — they walk ``pmix_server_globals``
  or fan out to a server-only framework:
  ``PMIx_server_IOF_deliver``, ``_IOF_flow_control``
  (``pmix_server_iof.c``);
  ``PMIx_server_dmodex_request`` (``pmix_server_dmodex.c``);
  ``PMIx_server_collect_job_info`` (``pmix_server_fence.c``);
  ``PMIx_server_setup_fork`` (``pmix_server.c``).  All four of those
  files have since had their own per-file pass (2026-08-24) and none of
  them added the screen, so this list is what the completed ``src/server``
  per-file review left open rather than something it has not reached yet.
* **Known must not get it** — ``PMIx_server_setup_application`` and
  ``_setup_local_support`` (``pmix_server_setup.c``), for the reason
  recorded above: a launcher calls them after ``PMIx_tool_init`` alone.
* **Almost certainly must not get it** — pure helpers over ``preg`` and
  hwloc, both of which ``pmix_rte_init`` stands up for every role, so
  they work correctly in a client today: ``PMIx_generate_regex``,
  ``_generate_regex2``, ``_parse_regex2``, ``_generate_ppn``,
  ``PMIx_server_generate_locality_string``, ``_generate_cpuset_string``,
  ``_generate_cpuset`` (``pmix_server_setup.c``), and
  ``PMIx_Store_internal`` (``pmix_server_dmodex.c``), which stores through
  ``pmix_globals.mypeer``'s gds module.
* **Must be left exactly as they are** — the two lifecycle sites in
  ``pmix_server.c``.  ``PMIx_server_init``'s check is the already-latched
  re-entry arm, and ``PMIx_server_finalize``'s guards the reference count;
  changing either one changes init/finalize semantics rather than closing
  a crash.

``PMIx_server_setup_fork`` is the one worth care: a launcher comes up
through ``PMIx_tool_init`` and reads as a server long before
``PMIx_server_init`` runs, so confirm no in-tree or PRRTE path calls it in
that window before adding the screen.

``PMIX_GET_POINTER_VALUES`` is honored by three shortcuts and nothing else
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found reviewing ``src/client/pmix_client_get.c`` (2026-08-22).

The attribute asks that "any pointers in the returned value point
directly to values in the key-value store".  ``process_request()`` obeys
it for the two requests it answers out of ``pmix_globals`` —
``PMIX_PROCID`` returns ``&pmix_globals.myidval`` and ``PMIX_RANK``
returns ``&pmix_globals.myrankval`` — and, inconsistently, not for
``PMIX_VERSION_NUMERIC``, which allocates.  Every other path, local hit
or server round trip, hands back an allocated copy the caller must
release.

Closing it is not a fix but a change to the ownership contract
``PMIx_Get(3)`` states, and one that has to answer a question the
shortcuts do not raise: a pointer into the datastore is only safe while
the datastore holds it, and ``gds/hash`` rewrites its tables on the
progress thread.  The two shortcuts are safe because they point at
process-lifetime globals.  Left as recorded behavior; the smaller,
separable piece is making ``PMIX_VERSION_NUMERIC`` agree with its two
neighbours.

The server and tool give back mypeer's second reference the long way
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found reviewing ``src/client/pmix_client.c`` (2026-08-22); what is left
belongs in ``src/server/pmix_server.c`` and ``src/tool/pmix_tool.c``.

``pmix_globals.mypeer`` carries a second reference on those two roles,
taken when ``pmix_client_globals.myserver`` is pointed at that same
object.  Both give it back by releasing ``pmix_globals.mypeer`` a second
time rather than by releasing ``myserver``, which is the pointer that
took it.  That works, but it costs the tool a ``myserver_is_mypeer``
flag captured before ``pmix_rte_finalize()`` purely to suppress a
release that would otherwise be a third one, and it left both roles with
``pmix_client_globals.myserver`` naming freed memory afterwards.  The
dangling pointers are fixed; the shape is not.

Releasing ``myserver`` *before* ``pmix_rte_finalize()`` — which is what
the client already does — would cover both cases with no flag and no
ordering subtlety: the object stays alive on the ``mypeer`` reference
throughout the runtime teardown, and ``rte_finalize`` then frees it.
Not done here because it reorders two finalize paths this review did not
cover, and confirming nothing in the ``rte_finalize`` chain reads
``pmix_client_globals.myserver`` wants more than a grep.

A locally fork/exec'd job gets none of its spawn's IOF directives
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found reviewing ``src/client/pmix_client_spawn.c`` (2026-08-22); the fix
belongs mostly in ``src/common/pmix_pfexec.c``.

``PMIx_Spawn_nb`` has three dispatch paths.  The server-role one and the
client/tool one both call ``pmix_server_spawn_parser()`` to read the
request's output directives into the caddy's ``channels``, ``flags`` and
``inherit_iof``; the server path then hands them to
``pmix_server_process_iof()`` and the client path copies ``flags`` onto
the namespace when the reply names it.  The fork/exec path a
disconnected launcher takes never calls the parser at all, and
``pfexec``
reads none of those fields.  The namespace object it creates keeps the
zeroed ``iof_flags`` its constructor gave it, and
``pmix_iof_write_output()`` formats from exactly that.

So ``PMIx_Spawn`` with ``PMIX_IOF_TAG_OUTPUT`` (or any other output
directive) produces tagged output from a connected launcher and
untagged output from a disconnected one.  ``PMIx_Spawn(3)`` says the
fork/exec fallback exists "allowing tools to maintain a single code
path for both the connected and disconnected cases", which is exactly
what this breaks.

Two pieces are needed and they are not the same size.  Parsing on that
path is one line, and belongs where the other two paths do it.  Acting
on the result is a ``pfexec`` question: it knows the namespace
immediately, so unlike the client path it could honor an
output-to-file directive rather than having to drop it — which makes
this a small feature rather than a transcription.  Left undone here
because adding the parse alone would produce a value nothing reads, and
because there is no in-tree way to exercise a disconnected launcher's
fork/exec output end to end.

A tool is sent key-deletion notices it has no receive posted for
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found reviewing ``src/client/pmix_client.c`` (2026-08-22); the fix
belongs in ``src/tool/pmix_tool.c``.

``pmix_server_notify_deleted()`` walks ``pmix_server_globals.clients``
and sends ``PMIX_PTL_TAG_DATA_DELETE`` to every peer there that is not
finalized and is not earlier than 7.0.0.  A tool that attached to the
server is in that array and reports its real version, so it is sent the
notice — but only ``PMIx_Init`` posts a receive for that tag.
``PMIx_tool_init`` does not, so the message reaches
``pmix_ptl_base_process_msg()`` with nothing waiting for it, is
discarded, and raises a ``PMIX_ERROR`` event on the way out.

Two things are wrong with that, and they want different fixes.  A tool
that has cached a key another process then deleted goes on answering
with the stale value, which is the whole reason the notice exists; that
argues for the tool posting the same receive and using the same handler.
Separately, a server should not be telling a peer something it cannot
have arranged to hear; the version test is the wrong screen for a role
that never posts the receive at all.

Nothing forwards a global-syslog request to a gateway
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/plog`` review (2026-08-20).  Half of it was
fixed; the other half needs a design decision.

``PMIX_LOG_GLOBAL_SYSLOG`` means "record this in the system-wide
syslog", and only a gateway server is meant to emit it — the message is
supposed to travel to the gateway node and be written there.  The
``syslog`` module implements the gateway half: if it is a gateway it
writes locally, and if it is not it declines.  Nothing implements the
other half.  There is no transport that moves the request to a gateway,
so on a peer that is not one the entry has nowhere to go.

Before the review this was invisible, because the module returned
``PMIX_SUCCESS`` regardless and the caller was told the message had been
logged.  It now declines the entry, so the framework reports
``PMIX_ERR_NOT_AVAILABLE`` (or ``PMIX_ERR_PARTIAL_SUCCESS`` alongside
another channel that did work) and a *client* falls back to its own
modules — which will decline for the same reason.  The failure is
honest now, and still a failure.

Closing it means choosing between two designs and is not a bug fix:
either the request is relayed to the gateway over the existing
server-to-server path, which needs the routing to exist and raises the
question of what a client should be told while it is in flight, or
``PMIX_LOG_GLOBAL_SYSLOG`` is documented as gateway-only and the
attribute's description in ``include/pmix_common.h.in`` is corrected to
say so.  Note that ``pmix_log_host_only`` and the host's ``log2`` entry
point already give a resource manager a way to take the request and do
the forwarding itself, which may be the answer that needs no new PMIx
machinery at all.

A compressed blob's length prefix is taken on trust
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/pcompress`` review (2026-08-20) and left alone
deliberately.

Every component allocates the uncompressed length the blob's 4-byte
prefix claims, *before* inflating anything.  The prefix generally came
off a peer's wire, so a five-byte blob whose prefix reads ``0xFFFFFFFE``
asks for a four-gigabyte allocation, which then fails or succeeds and is
thrown away when the payload turns out not to decode.  The review closed
the case that was a memory error — a blob too short to hold the prefix
at all — but not this one, which is a resource question rather than a
correctness one.

The obvious guard is a maximum expansion ratio, and that is exactly why
it was not written: DEFLATE tops out near 1032:1 while zstd's is far
higher, so any single cap either fails to constrain zstd or rejects
legitimate zlib output.  A per-component cap is possible; whether it is
worth the interoperability risk is a policy decision, not a bug fix.
Note also that the caller has already read the whole blob into memory by
the time it gets here, so the amplification is bounded by what the PTL
was willing to accept.

A pcompress module's ``init()`` failure is fatal to library init
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Also from the ``src/mca/pcompress`` review, and dead code today.

The framework's documented stance is that having no compressor is not an
error: ``pmix_compress_base_select()`` returns ``PMIX_SUCCESS`` when it
selects nothing, and the base default no-op stubs stay in place.  But if
the winning module's ``init()`` fails, that error is returned, and
``pmix_init.c`` treats a non-``SUCCESS`` return from the select as fatal
to ``PMIx_Init``.  So a compression library that loads but cannot start
would take the whole library down, where an absent one is shrugged off.

Unreachable: no module in the framework implements ``init``, and every
one of the five leaves the slot ``NULL``.  It is recorded because the
first module that does implement it will inherit the inconsistency, and
the fix — degrade to the base default rather than fail — should be made
then, with a module to test it against.

An absent app-level value fails a child's launch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/pmdl`` review (2026-08-19), and narrowed rather
than closed.  ``pmdl/ompi``'s ``setup_fork`` treats a missing
``PMIX_PROCDIR``, ``PMIX_WDIR`` or ``PMIX_APP_ARGV`` as an error, and a
``setup_fork`` error fails ``PMIx_server_setup_fork`` and with it the
child's launch — so a host that registers a namespace without one of them
cannot start the job at all, however little the resulting envar matters
(``OMPI_MCA_initial_wdir`` is informational).

The review closed the case that was unambiguously wrong —
``PMIX_REINCARNATION``, which nothing in PMIx sets and no host is obliged
to provide, so its absence now means "zero restarts" rather than a failed
launch.  The three above are ordinary registration data that every host
in practice provides, so making them optional would be a behavior change
with nothing to test it against; it needs a decision about which of them
Open MPI genuinely requires.

Fabric inventory collection is a stub
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/pnet`` review (2026-08-19).  No pnet component
collects inventory.  ``opa``'s ``collect_inventory`` carries a comment
about searching the topology for OPA NICs and then returns
``PMIX_SUCCESS`` having added nothing; ``deliver_inventory`` returns
``PMIX_SUCCESS``.  ``nvd``'s goes one step further — it confirms that a
matching NIC exists on the node — but still reports no contents.  The
plumbing is real, so a host that asks for fabric inventory gets a
successful, empty answer rather than an error; treat the capability as
unfinished rather than working.

Finishing it needs a decision the review was not entitled to make: what a
fabric inventory record contains, and how a component that has nothing to
report says so.  The base has **no decline convention** on this path —
unlike ``allocate`` and ``setup_fork``, it ``PMIX_ERROR_LOG``\ s any
non-``PMIX_SUCCESS`` return and abandons the fan-out to every component
behind it — so today "nothing here" and "collected everything" are the
same answer.

The deprecated regex API cannot carry a length
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/preg`` review (2026-08-19), and narrowed rather
than closed by it.  ``pmix_preg_base_legacy_decode`` bounds every read
against an ``avail`` argument, but only ``unpack`` can supply a real one:
it knows how many bytes are left in the buffer.  Every other caller holds
a bare ``char *`` and passes ``SIZE_MAX``, because
``pmix_preg.parse_nodes(regexp, &names)`` has nowhere to put a length
without changing a signature that predates the current API.  ``gds/hash``
has the length in ``val->data.bo.size`` and still cannot hand it over.

The review made the tag test exact, which removed the case that a host
could trigger with ordinary data — a plain node list whose first node
begins with ``blob`` is no longer taken for a blob and walked past its
end.  What is left needs a caller-owned string that really does carry the
``"blob:"`` tag and is truncated behind it, which is not something a peer
can produce (those arrive through the bounded ``unpack``).  Closing it
properly means plumbing a length through the deprecated signatures, and a
caller that can do that is better off moving to ``pmix_regex2_t``.

* **``pmix_debug_threads`` cannot be turned on.**  The four lock macros
  emit distinct debug strings under ``if (pmix_debug_threads)``, but
  nothing in the tree ever assigns that variable — there is no MCA
  parameter for it and no other write outside its definition in
  ``src/threads/thread.c``.  The facility is reachable only by setting
  the variable from a debugger.  Registering an MCA parameter for it
  would cost a few lines; it was left alone because ``src/threads`` is
  semantics-frozen and this is a feature rather than a defect.  Until
  then, do not read a silent run as evidence the handshake was not
  exercised.

A shared session's segment has nothing that shares it
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found in the ``src/mca/gds/shmem3`` re-review (2026-08-21).  The
component places a job's session segment deliberately *outside* the
per-job address arena, because a session is meant to outlive the job that
first described it — ``pmix_mca_gds_shmem3_component.sessions`` holds a
reference, and ``job_destruct()`` releases the whole arena, so a shared
session's segment inside it would be unmapped under a live holder.

The sharing that guards against cannot happen today: **nothing ever
appends to that list**.  It is permanently empty, both searches in
``pmix_gds_shmem3_get_session_tracker()`` are unreachable, and a job's
session object is created by ``job_construct()`` and dies with it — so
two jobs in the same session each build their own copy of the session
data rather than mapping one segment.

The placement was left as it is, because it is what the code would need
the moment a session really were shared, and moving it would have to be
undone.  What is deferred is the other half: deciding when a session
tracker is registered on the component, and what its lifetime is once
more than one job holds it.  Do not read the arena comment as evidence
that sharing works.

A server-wide envar hook that nothing fills
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Found reviewing ``src/server/pmix_server.c`` (2026-08-23).
``pmix_server_globals.genvars`` is declared as "argv array of envars given
to me for passing to all clients", is statically initialized to NULL, and
is read in exactly one place — ``setup_fork_body()`` replays it into every
child's environment.  **Nothing in the tree ever writes it.**  There is no
directive, no API argument and no environment variable that reaches it, so
a host has no way to say "add these to every client I fork" and the replay
loop is dead code.

This is an unbuilt feature rather than a defect, which is why it is here
rather than fixed: closing it means choosing the interface.  The obvious
candidate is ``PMIx_server_register_resources`` — it already carries
host-supplied job-wide information into ``pmix_server_globals.gdata`` —
with the ``PMIX_SET_ENVAR`` / ``PMIX_ADD_ENVAR`` / ``PMIX_PREPEND_ENVAR``
family that ``src/common/pmix_pfexec.c`` already parses for a spawn.  That
also decides the harder half: whether a later registration replaces or
appends, and whether ``PMIx_server_deregister_resources`` has to take an
envar back out of children that have already been forked (it cannot).
The comment above ``setup_fork_body`` used to assert that the registration
path sets it; that has been corrected, and ``src/server/AGENTS.md`` says
the read site is not evidence of a writer.

Coverage gaps
-------------

* **Nothing in CI can detect a data race.**  The sanitizer job builds
  with ``-fsanitize=address``; ASan finds no races, and there is no
  ThreadSanitizer build anywhere.  This became visible closing the
  ``PMIx_server_init`` ordering question (2026-08-23): the library's
  central threading invariant — that ``pmix_globals``, the ``gds``
  tracker lists and ``pmix_server_globals`` are touched only from the
  progress thread — is held up by reading code, and every claim that a
  particular path honors it is argued rather than measured.  ``gds/hash``
  takes no lock at all, so a violation is heap corruption rather than a
  stale read, and the interleavings that would expose one are measured in
  microseconds and need two processes.  A TSan configuration exercised by
  the tool and server tests is the missing instrument; until there is
  one, treat "this runs on the progress thread" as an assertion to be
  re-checked by hand whenever the code around it moves.

* **The out-of-memory and finalize-race arms of the server switchyard's
  host callbacks.**  ``op_cbfunc``, ``op_cbfunc2`` and ``resop_cbfunc``
  in ``src/server/pmix_server_switchyard.c`` each have two arms that do
  not thread-shift.  Both were fixed to release the caddy they own, but
  neither is reachable from a unit test without fault injection — a
  failed allocation, or the progress thread stopping while a host
  completion is in flight.  The same gap now covers the 32 dispatch arms
  that answer ``PMIX_ERR_NOMEM`` when ``PMIX_GDS_CADDY`` cannot allocate,
  and the ``PMIX_ERR_NOMEM`` arm of ``PMIX_SERVER_QUEUE_REPLY``
  (2026-08-24): both replaced a NULL dereference that killed the server,
  and both need an allocator hook to reach.  An allocation-failure
  injection facility — a debug-build counter that makes the *n*\ th
  ``pmix_malloc`` fail — would close a good part of this directory's
  untestable set at once; it is the single highest-value piece of test
  infrastructure the review has wanted.
* **A multi-namespace ``PMIx_Disconnect`` is not reached by ``make
  check``.**  ``test/test_cd.c`` — what the ``--test-connect`` runs
  drive — disconnects a process from its *own* namespace, so the loop in
  ``disconnect_cbfunc()`` that drops the other participants' cached data
  never executes.  Reaching it needs live processes in two namespaces,
  and therefore a real launcher, for the reason in the next entry.  The
  path is verified today by running ``examples/dynamic.c`` under a PRRTE
  DVM with ``PMIX_MCA_gds_base_verbose=1`` forwarded to the clients, so
  the ``GDS DEL NSPACE`` lines can be seen coming from a client rather
  than only from the server's deregistration.  Closing this means a test
  that stands up a server and two client namespaces in-process, in the
  white-box style of ``test/unit/server_fence.c``.
* **``test/simple/simptest`` cannot host a spawn.**  Its ``spawn_fn``
  calls ``PMIx_server_setup_application()``, a thread-shifting API, and
  then waits on the result — but ``spawn_fn`` is the host callback and
  runs *on* the server's progress thread, so the fake launch deadlocks.
  Spawn coverage therefore lives in the multi-node suite rather than in
  ``make check``.  Do not add a ``run_*.pl`` that spawns until that path
  is made asynchronous.
* **The reply handling of the ``PMIx_Compute_distances`` pair is not
  covered anywhere,** and cannot be: two of the three paths are
  allocation failures and the third needs a server reply whose declared
  count exceeds its payload.  Reaching the receive path at all requires
  a client whose *local* hwloc computation failed.
* **Two fixes found in ``src/tool`` ship without regression tests**
  because the conditions cannot be arranged in ``make check``: the
  SIGCONT handler for forwarded stdin needs a tty, and the
  late-finalize-reply guard needs a server that answers the finalize
  handshake more than five seconds late.  Only the second is still in
  ``src/tool``; the stdin read event and its SIGCONT handler moved to
  ``src/common/pmix_iof.c`` when the tool was given the library's
  forwarding instead of its own, and are untested there for the same
  reason.
* **The debugger-wait teardown in ``PMIx_Init`` is not exercised.**
  Reaching it needs the debugger-stop key set on the namespace *and* the
  ``PMIX_READY_FOR_DEBUG`` notification to then fail, which no test
  environment can arrange without fault injection.
* **The process-set and resolve examples have never been leak-validated.**
  Both hang in the test environments used so far, so valgrind is killed
  before ``PMIx_Finalize`` runs and the report is inconclusive.
* **``pps`` has never been validated against a live process-table
  server.**  Its no-connect paths are covered by the tools smoke test;
  the proc-table rendering is not.
* **The compressed half of ``preg`` is invisible to a build with no
  compression library.**  ``preg/compress`` disables itself when
  ``pcompress`` has no module, so on such a build — a stock macOS
  developer tree among them — ``raw`` wins every encode, and neither the
  ``blob:`` framing in ``preg_base_legacy.c`` nor the ``compress``
  component is ever reached by a round trip.  ``test/unit/preg`` covers
  the framing with hand-built vectors, which is what a bounded decoder
  can be tested with, but the encode-decode pair only meet on a build
  that can compress: ``test_legacy_large`` says which encoding it got, so
  a thin run reports itself rather than passing quietly.  The real round
  trip was run under ``contrib/dockerswarm`` (Linux, all four
  compressors), where the 5000-node case does take the blob path.
* **The ``psensor/file`` drop-count and baseline semantics have no
  automated check.**  The fix stops a monitor from alerting before it has
  ever recorded a miss — a request with no ``PMIX_MONITOR_FILE_DROPS``
  used to trip on the first sample of a perfectly healthy file.
  Distinguishing the fixed behavior from the broken one takes a monitor
  with a *zero* drop allowance watching a file that is being kept fresh,
  and zero tolerance means any scheduling hiccup or coarse filesystem
  timestamp granularity fails it.  The heartbeat drop allowance is
  covered instead (``run_monitor.pl``'s capped monitor), because a
  request can be given an allowance no run can spend; there is no
  equivalent for "must not alert on the first look".
* **A handshake-model psec module blocks the progress thread for as long
  as its peer takes to answer.**  ``PMIX_PSEC_SERVER_HANDSHAKE_IFNEED``
  runs inside the ``ptl`` connection handler, on the progress thread,
  with the socket deliberately still in blocking mode; a peer that
  connects and then stops writing pins that thread until the socket
  errors out.  This is intrinsic to the way ``ptl`` sequences the
  connection handshake rather than anything ``psec`` chooses, and today
  the only handshake-model module is ``dummy_handshake``, which is
  test-only.  It becomes a real availability question the moment a
  genuine one is written, and the fix belongs in ``ptl`` — a timeout on
  the handshake exchange, the way ``handshake_wait_time`` already bounds
  the connect-ack.  Recorded here so a new mechanism does not inherit it
  silently.
* **``psec/munge``'s failed-encode path is still not executed.**  The
  rest of the component now is: ``test/unit/psec_credentials.c`` drives
  every *active* credential module rather than a fixed list, so on a host
  with libmunge and a live ``munged`` it examines ``munge`` on the same
  terms as ``native``.  The PRRTE tree's ``contrib/slurmswarm`` image is
  such a host — it installs ``libmunge-dev`` before it builds PMIx, so
  the component is compiled, and its entrypoint starts ``munged`` — and
  the suite passes 62/62 there, valgrind-clean, with the ``info[n]``
  defect confirmed to segfault it when reinstated.  What no test reaches
  is the branch where ``munge_encode`` *fails* midway through a refresh,
  which is where the dangling ``mycred`` lived; provoking it means making
  ``munged`` fail on demand between two ``PMIx_Get_credential`` calls.
* **The plog ``smtp`` component has never been run.**  It builds only
  where libesmtp is present, and exercising it needs a reachable SMTP
  server on top of that, so the fixes from the ``src/mca/plog`` review
  (2026-08-20) — an uninitialized ``message_status_t``, a ``crnl()``
  that emitted LF where it meant CR, a message callback that ended the
  message before the body when no prefix was configured — were verified
  by reading and compile-checked with ``--enable-test-build`` against
  the shim header, not by sending mail.  The shim makes every libesmtp
  call a stub, so a test-build tree cannot exercise them either.  The
  first real user of this component should expect to find more.
* **Nothing exercises the pnet fabric calls.**  ``register_fabric``,
  ``update_fabric`` and ``deregister_fabric`` are wired all the way
  through ``src/mca/pnet/base``, but no shipped component implements any
  of them, so ``pmix_pnet_globals.fabrics`` is never populated in a stock
  build and every base fabric call ends in ``PMIX_ERR_NOT_SUPPORTED`` /
  ``PMIX_ERR_NOT_FOUND`` / ``PMIX_ERR_BAD_PARAM``.  Covering it means
  writing a component that claims a plane, which is the decision recorded
  above.  (``test/unit/server_fabric`` covers the server-side
  ``PMIx_Compute_distances`` handler, not this path.)
* **``pnet/simptest``'s end-to-end launch path is not in** ``make check``.
  ``test/unit/pnet_simptest_map`` drives ``allocate`` through
  ``PMIx_server_setup_application`` against a topology file it writes, but
  the other half — a client fetching ``PMIX_FABRIC_ENDPT`` by rank and
  ``PMIX_FABRIC_COORDINATES`` at the node level — still has to be run by
  hand, because ``test/simple/simptest`` generates its node map from the
  local host and so needs a topology file naming that host.  See
  "Running it" in ``src/mca/pnet/simptest/AGENTS.md``.

* **The TSD finalize-ordering fix has no automated test.**
  ``pmix_tsd_keys_destruct`` deletes the process's pthread keys, so it
  may only run once every other thread is joined; it used to run six
  lines above the ``pmix_progress_thread_stop`` in ``pmix_rte_finalize``
  that joins the progress thread, and has been moved below it.  Neither
  half of what that fixed is testable from a unit program: reaching a
  deleted key needs the progress thread to print a process name inside a
  window a few calls wide, and the key-slot leak needs the same window
  and then shows up only as a ``PTHREAD_KEYS_MAX`` exhaustion many
  init/finalize cycles later.  ``test/unit/threads_primitives.c`` covers
  the registry's own contract — including that a destructor is *not* run
  for a key the finalizing thread never set a value for — but the
  ordering itself is held only by the comment at the call site and by
  ``src/threads/AGENTS.md``.

* **``gds/shmem3`` gets no coverage at all on macOS, of any kind.**  Its
  ``configure.m4`` gates on a 64-bit non-Apple host with no
  ``|| test "$pmix_testbuild" = "1"`` escape, so a Mac does not even
  *compile* it — ``--enable-test-build`` does not help, and a change made
  there has not been compile-checked until it has been built on Linux.
  ``test/unit/gds_datastore``'s ``test_shmem3_job_segment()`` asks
  ``pmix_gds_base_assign_module()`` for the component by name and prints
  ``SKIP`` where it is absent, so the case is honest rather than missing,
  and on Linux it drives the segment build end to end in one process.
  Everything beyond one process — a fence, a second modex generation, a
  client attaching at a fixed address, and every cross-node fetch — is
  reachable only through ``contrib/dockerswarm/run-gds-tests.sh``.
* **The client-side tombstone generation fix is not what**
  ``examples/delete_key.c`` **discriminates on.**  The example now
  re-publishes a deleted key and requires every rank to read the new
  value back, which pins the documented behavior — a deletion is not
  permanent — but it passes both before and after the fix: on a client
  the keyed get misses locally and is answered by the server, which never
  had the bug.  Catching the client-side half needs a NULL-key read that
  actually reaches the modex, which takes the local table to miss for the
  rank or an explicit ``PMIX_REMOTE`` scope, and nothing arranges that
  today.

Not defects — by design
-----------------------

These look like bugs and are not.  They are recorded so that they are
not "fixed" by a later reader.

* **A ``pmix_lock_t`` whose ``status`` reads ``PMIX_ERR_INIT`` is
  reporting a missing assignment, not an init failure.**  Both
  ``PMIX_CONSTRUCT_LOCK`` and ``PMIX_LOCK_STATIC_INIT`` seed ``status``
  with ``PMIX_ERR_INIT``.  A waiter reads it only after a handler has
  woken it, so every handler on a path whose waiter reads ``status`` must
  assign it — and nothing enforces that, which makes the default the
  thing that decides how a violation presents.  ``PMIX_SUCCESS`` would
  turn a forgotten assignment into a confident wrong answer;
  ``PMIX_ERR_INIT`` makes it loud.  ``PMIX_CONSTRUCT_LOCK`` previously
  assigned nothing at all, and since the lock usually lives in a
  ``PMIX_NEW``'d caddy — ``malloc``, not ``calloc`` — the alternative was
  heap garbage.  If one of these turns up in a bug report, look at the
  handler that woke the lock and check its error paths before believing
  the status.

  The sweep behind this is worth not redoing.  Poisoning ``status`` and
  warning on it in ``PMIX_WAIT_THREAD`` fires ~86,000 times across
  ``make check``, which looks damning and is not: those callers read
  ``cb->status``, the *caddy's* own field, and never touch the lock's.
  Grepping ``lock.status`` and ``lock->status`` specifically narrows it
  to about nine sites in the library, every one woken through an
  ``opcbfunc`` that assigns ``status`` first.  The two spellings are a
  character apart and only instrumentation separates them.

* **``PMIX_ACQUIRE_THREAD`` / ``PMIX_RELEASE_THREAD`` having no callers
  in** ``libpmix`` **does not make them dead code.**  An in-tree grep
  finds only ``test/unit/threads_primitives.c``, and the live handshake
  everywhere in the library is ``WAIT``/``WAKEUP`` at about 120 sites
  each — which reads as a retirement candidate and is not one.  These
  headers are installed under ``$(pmixincludedir)/src/threads``, and
  PRRTE uses the pair: ``src/runtime/prte_locks.h`` includes
  ``src/threads/pmix_threads.h`` to declare ``prte_init_lock`` as a
  ``pmix_lock_t``, and ``prte_init()``, ``prte_finalize()`` and
  ``src/prted/pmix/pmix_server_notify.c`` guard ``prte_initialized`` with
  it across eleven call sites (checked against openpmix/prrte master,
  2026-08-20).  Removing them breaks PRRTE's build, and changing their
  semantics changes PRRTE's behavior.

  This is worth keeping precisely because the in-tree evidence points the
  wrong way, and because unused-looking API is where documentation rots:
  ``src/threads/AGENTS.md`` had described ``ACQUIRE_THREAD`` as *not*
  implying that it returns holding the mutex, when that pair is exactly a
  held-mutex critical section, so a reader who believed the guide and
  mixed the pairs would unlock a mutex they do not hold.  That is
  corrected, and the unit test now covers the pair.

* **The function-pointer cast in ``pmix_thread_start`` is deliberate.**
  It hands a ``void *(*)(pmix_object_t *)`` to ``pthread_create``, which
  wants a ``void *(*)(void *)`` — formally a call through an incompatible
  function pointer type.  GCC's ``-Wcast-function-type``, which is on
  through ``-Wextra -Werror``, treats any pointer parameter as matching
  any other and does not diagnose it, and every ABI PMIx supports passes
  a ``void *`` and a struct pointer identically.  Only clang's opt-in
  ``-Wcast-function-type-strict`` and UBSan's ``-fsanitize=function``
  object, and neither is used by the build or by CI — the sanitizer job
  in ``.github/workflows/builds.yaml`` is ASan only.  If one of those is
  ever turned on, the fix is a small trampoline that takes ``void *`` and
  calls ``t_run``, not a change to ``pmix_thread_fn_t``.

* **A statically initialized mutex is not ``ERRORCHECK`` even in a debug
  build.**  ``pmix_mutex_construct`` sets ``PTHREAD_MUTEX_ERRORCHECK``
  under ``PMIX_ENABLE_DEBUG`` so self-deadlocks and double-unlocks abort
  loudly, but ``PMIX_MUTEX_STATIC_INIT`` expands to
  ``PTHREAD_MUTEX_INITIALIZER`` — an ordinary mutex — because there is no
  static spelling of a mutex attribute.  This is a property of pthreads,
  not an oversight, and it cannot be fixed without giving up static
  initialization.  The consequence to remember is diagnostic: a clean
  debug run over a file-scope lock says nothing about whether it can
  deadlock.

* **``pmix_mutex_construct`` ignoring ``pthread_mutex_init``'s return is
  not an unchecked error.**  The constructor returns ``void``, so there
  is nothing it could do but abort, and on both glibc and macOS
  ``pthread_mutex_init`` with a valid attribute allocates nothing and
  cannot fail.  POSIX permits ``EAGAIN``/``ENOMEM``, so if a platform
  that can really fail it ever appears, add a debug-only ``perror`` and
  ``abort`` matching the ``EDEADLK``/``EPERM`` checks in the same file
  rather than trying to report it upward.

* **A** ``pcompress`` ``compress_string`` **that does not screen its
  argument for NULL is not a missing guard.**  All five implementations
  hand the pointer straight to ``strlen``, which reads as an asymmetry
  now that both *decompress* entry points screen for NULL — but the two
  directions take different data.  A decompressor is handed a length and
  a buffer a peer declared; a compressor is handed a string the caller
  just built and owns.  No caller in the tree can produce a NULL there,
  and a component answering ``false`` for one would report "I declined
  to compress" for what is really a caller bug, hiding it.  Add the
  screen only alongside a caller that can actually pass NULL.

* **The fence modex bucket reported as a libpmix leak belongs to the
  host.**  ``pmix_server_fence`` unloads the assembled bucket with
  ``PMIX_UNLOAD_BUFFER`` and hands the bare pointer to
  ``pmix_host_server.fence_nb``; ownership transfers on the call, and the
  request direction carries no ``(release_fn, release_cbdata)`` pair the
  way the reply direction does.  We cannot free it on return — the host
  parks it and reads it from another thread — so a host that does not
  free it leaks about 128 bytes per collecting fence.  In a valgrind run
  against a launcher this looks exactly like a libpmix leak and nothing
  else does: the allocation is ``pmix_server_collect_data``, the stack
  bottoms out in ``progress_engine``, and there is no ``prte_`` frame
  anywhere in it.  Both in-tree hosts (PRRTE and ``test/simple/simptest``)
  currently leak it.  See ``src/server/AGENTS.md``.

* **An event-caching block that assigns ``cd->nondefault`` inside its**
  ``0 < ninfo`` **guard is not dropping a flag.**  It reads as though a
  non-default event carrying no info would be parked as a default one
  and then delivered to default handlers that should not see it.  It
  cannot happen: every path that sets ``chain->nondefault`` reads it out
  of a ``PMIX_EVENT_NON_DEFAULT`` directive in the info array, so the
  flag is only ever true when there is info to have carried it.  Moving
  the assignment out of the guard is equivalent, not a fix — two of the
  three caching blocks now assign unconditionally only because the
  guarded form kept being re-reported.  The third, in
  ``pmix_notify_server_of_event``, still assigns inside the guard and is
  equally correct.
* **``pmix_srand()`` copies the seeded state into a file-static buffer**
  (``src/util/pmix_alfg.c``).  It looks like a footgun, but it is the
  only way to seed the global that ``pmix_random()`` reads, and the unit
  test deliberately holds down that behavior.  ``pmix_random`` is unused
  in-tree and the real ``pmix_srand`` callers use their own buffers, so
  the "last writer wins" hazard is not reachable.  Do not drop the copy
  without giving ``pmix_random`` another way to be seeded.
* **``wait_signal_callback()`` reads the child-list size without a
  lock** (``src/common/pmix_pfexec.c``).  This is a formal data race,
  kept on purpose: the child is appended to the list before the ``fork()``
  that can generate its ``SIGCHLD``, so the count cannot be stale for a
  child that matters, and removing the guard would make pfexec call
  ``waitpid(-1)`` on every ``SIGCHLD`` in a process with no children of
  its own.  If it is restructured, replace the read with a counter
  maintained across all six add/remove sites.
* **A failed ``pmix_rte_init`` returns without unwinding.**  It emits a
  diagnostic and returns, deliberately leaving the frameworks, globals
  and event base it had already brought up, because a failed
  ``PMIx_Init`` aborts the process.  New failure points should follow the
  same pattern.
* **``get_job_data`` returning success with an empty buffer is safe.**
  The requesting client initializes its reported status to
  ``PMIX_ERR_NOT_FOUND`` and overwrites it only if a value turns up, so
  an empty success degrades to not-found at the requester.
* **``pmix_server_job_ctrl`` creating a namespace for an unknown target
  is intentional.**  A job-control request may name a job this server has
  not been told about yet, and the epilog directives need somewhere to
  hang; the entry is reused when registration arrives.
* **The plog router builds its per-request channel list out of the
  global module wrappers.**  ``pmix_plog_base_log`` appends the
  ``pmix_plog_base_active_module_t`` objects that live in
  ``pmix_plog_globals.actives`` onto a local ``pmix_list_t``, so the
  list links and the ``added`` flag are written into shared state.  It
  reads as a re-entrancy bug waiting to happen, and it would be one — a
  module whose ``log`` reached ``pmix_plog_base_log`` synchronously
  would relink the objects out from under the loop walking them.  No
  module does: ``stdfd`` hands off to ``PMIx_server_IOF_deliver``, which
  posts an event and returns, and ``pmix_show_help`` thread-shifts
  rather than calling down inline.  Copying the wrappers per request
  would cost an allocation on every log call to defend against a caller
  that does not exist; the invariant is documented in
  ``src/mca/plog/AGENTS.md`` instead.  Enforce it there, not here.
* **``pnet/opa`` and ``pnet/nvd`` ship no ``configure.m4``, on purpose.**
  A component with no ``configure.m4`` is configured by the MCA machinery
  itself and therefore builds unconditionally, which is what these two
  want: neither links anything nor needs an SDK — they read info
  attributes hwloc already recorded and set environment variables — and
  whether there is work to do is a property of the machine the *daemon*
  runs on, which only ``component_open`` is in a position to know.  The
  files they used to have asked that question at build time and got it
  wrong in both directions (``nvd``'s was hardwired off, ``opa``'s always
  succeeded).  The one visible consequence is that ``configure``'s summary
  no longer prints ``Transports / NVIDIA|OmniPath`` lines.  Do not restore
  them.
* **``transports_print`` in ``pnet/opa`` type-puns a ``uint64_t`` through
  ``unsigned int *``.**  That is undefined behavior in general and is not
  a live defect here: the whole tree is built ``-fno-strict-aliasing``
  (``config/pmix_setup_cc.m4`` adds it wherever the compiler takes it).
  The surrounding arithmetic is deliberately width-independent, and the
  byte-order dependence of the result does not matter because the key is
  generated once on the lead server and shipped as a string.  Do not copy
  the idiom into new code, and do not "fix" it as a bug.
* **A node in ``pnet/simptest``'s topology file that the job's node map
  does not name is silently skipped.**  ``allocate`` matches each config
  line against the ``PMIX_NODE_MAP`` by exact name, which is intrinsic to
  a file that describes the real nodes the RM placed the job on.  The
  converse is *not* silent: a node the job was placed on that the file
  fails to describe draws a ``node-not-found`` ``show_help`` naming it and
  the request is then declined — declined, rather than failed, because a
  hard error out of ``allocate`` aborts the base's fan-out for every other
  component too.
* **``psec/dummy_handshake`` sends its length and status words as raw
  host-format ``size_t`` / ``pmix_status_t``.**  That means it only
  interoperates between peers of identical width and endianness, which
  would be a wire-format defect in a real mechanism.  It is not one here:
  the component exists solely to exercise the ``ptl``/``psec`` handshake
  plumbing, is built only under ``--enable-dummy-handshake``, and is
  documented as not a pattern to copy.  Do not "fix" it by inventing a
  wire encoding for a test harness.
* **``pmix_psec_base_select`` sets ``pmix_psec_globals.selected`` before
  it can fail.**  A select that ends with an empty actives list returns
  ``PMIX_ERR_SILENT`` with the flag already true, so a second call would
  return ``PMIX_SUCCESS`` over an unusable framework.  There is no second
  call: ``pmix_init.c`` invokes it once and treats the failure as fatal
  to library init.  Left as-is rather than adding a rollback for a path
  that cannot be re-entered.
* **The bare ``atomic_bool`` fields in ``pmix_globals_t``** are correct,
  merely inconsistent with the typedefs used elsewhere, and the
  ``PMIX_C_HAVE_*`` defines in the installed ``pmix_config.h`` are now
  always ``1`` but are retained in case an out-of-tree consumer tests
  them.
