Specification: Tracking Local Collective Participation
======================================================

.. note::

   This is a *design specification*. It states how the PMIx server is
   required to track local participation in collective operations, and
   in particular how it must account for participants whose connections
   are lost. It documents the intended behavior and the invariants that
   any implementation must preserve. The companion implementation plan
   lives alongside this document in the same directory.

Scope
-----

This specification covers the server-side tracking of *local*
participation in the collective operations that PMIx assembles on a node
before forwarding to the host environment:

* ``PMIx_Fence`` / ``PMIx_Fence_nb`` (``PMIX_FENCENB_CMD``)
* ``PMIx_Connect`` / ``PMIx_Connect_nb`` (``PMIX_CONNECTNB_CMD``)
* ``PMIx_Disconnect`` / ``PMIx_Disconnect_nb`` (``PMIX_DISCONNECTNB_CMD``)
* ``PMIx_Group_construct`` / ``_destruct`` (``PMIX_GROUP_CONSTRUCT_CMD``,
  ``PMIX_GROUP_DESTRUCT_CMD``)

It does **not** cover the global (inter-node) phase, which is the
responsibility of the host environment, nor the wire encoding of the
collected data, which belongs to ``bfrops``.

Background: how a collective is assembled
-----------------------------------------

A collective names a set of participating processes. Some of those
processes are *local* — clients (and their clones) connected to this
server — and some may be remote. The server's job in the local phase is
to wait until **every local participant has contributed** its call, then
pass the assembled operation up to the host for global completion.

The server represents each in-flight collective with a *tracker*. For
fence/connect/disconnect this is ``pmix_server_trkr_t`` (defined in
``src/include/pmix_globals.h``); group operations use the parallel
``grp_block_t`` / ``grp_trk_t`` structures in
``src/server/pmix_server_group.c``. Trackers live on the server-global
lists ``pmix_server_globals.collectives`` and
``pmix_server_globals.grp_collectives`` respectively.

A tracker records, among other things:

* the participant list (``pcs`` / ``npcs``) as originally supplied;
* the *expected* number of local participants (``nlocal``);
* a list of the contributions received so far (``local_cbs``, a list of
  ``pmix_server_caddy_t``, one entry per contributing peer — including a
  separate entry for each fork/exec'd clone of a participating rank);
* whether the tracker *definition* is complete (``def_complete``: every
  participating namespace has been registered so ``nlocal`` is known to
  be final);
* whether the operation has already been handed to the host
  (``host_called``).

When a local client calls the collective, the server locates or creates
the matching tracker and appends the caller's caddy to ``local_cbs``.

The counter-based completion test and its vulnerability
-------------------------------------------------------

Historically, local completion has been decided by comparing a count of
contributions against the expected local count. In the current code the
test appears (in several places) as:

.. code-block:: c

   if (trk->def_complete &&
       pmix_list_get_size(&trk->local_cbs) == trk->nlocal) {
       /* local phase complete -- forward to host */
   }

That is: *the number of contributions received equals the number of
local participants expected.* This is a **cardinality** test — it counts
contributions and expected participants, but it does not record *which*
specific participants have contributed.

Separately, when a client's connection is lost, the server must account
for its departure so that a collective it was expected to join does not
hang forever. The lost-connection handler
(``lost_connection`` in ``src/mca/ptl/base/ptl_base_sendrecv.c``)
currently, for each tracker the departing peer belongs to:

#. decrements the expected count (``--trk->nlocal``); and
#. removes any contribution the peer had already made from
   ``local_cbs`` (matched by process name).

Because participation is tracked only as a count, these two adjustments
race against the ordinary participation path. The failure has two
distinct symptoms.

**1. Early completion.** A cardinality test cannot tell the difference
between "this participant has not called yet" and "this participant
called and then died." If the expected count is reduced for a
participant whose contribution was already counted, the remaining live
participants can drive the count to the (now lowered) threshold before
they have all actually called in. The collective completes early and is
forwarded to the host with a subset of the intended local contributions.

   *Illustration.* A fence expects three local participants: ranks 0, 1,
   and 2. Rank 0 calls the fence; its contribution is counted. Rank 0
   then dies before ranks 1 and 2 call. In a pure counter model, rank
   0's contribution remains counted while the expected total is
   decremented to two; when ranks 1 and 2 call, the count reaches the
   lowered threshold of two and the fence completes — even though it was
   supposed to also include the (still-live-at-call-time) semantics of
   all three, and rank 2 may not yet have called at all.

**2. Loss of already-delivered data.** For a data-collecting fence, a
participant's contribution *is* its modex data. Discarding a departed
participant's contribution after it was delivered removes that data from
the assembled collective, so the surviving participants receive an
incomplete result — a silent correctness error, not merely a timing one.

The root cause in both cases is the same: **participation is tracked as a
count rather than by participant identity.** A count cannot answer the
one question the lost-connection handler must ask — *"has this specific
participant already contributed?"*

Required model: track participation by identity
-----------------------------------------------

The server must track local collective participation **by participant
identity**, not by a bare counter. Conceptually, each tracker maintains
three sets of local participants:

``expected``
    The local participants the collective is waiting on. Established as
    namespaces register and as the participant list is resolved against
    the set of local processes (including ``PMIX_RANK_WILDCARD``
    expansion). A participant enters ``expected`` when it is known to be
    a local member of the operation.

``contributed``
    The local participants that have delivered their call for this
    collective, together with whatever data or info that call carried.
    A participant enters ``contributed`` exactly once per required
    contribution.

``departed``
    Local participants that were lost (connection dropped, abnormal
    termination) **before** contributing. A participant enters
    ``departed`` only if it was expected and had not already
    contributed.

Completion rule
~~~~~~~~~~~~~~~

The local phase is complete when the tracker definition is complete
(``def_complete``) **and every expected participant has been accounted
for** — that is, each expected participant is either in ``contributed``
or in ``departed``:

.. code-block:: text

   local_complete  ==  def_complete
                       AND  (contributed ∪ departed)  ⊇  expected

Equivalently, the collective is still pending as long as at least one
expected participant has neither contributed nor departed.

This must be the **single** completion predicate used everywhere the
question "is this collective locally complete?" is asked. Today that
question is asked, with a hand-rolled cardinality test, in at least six
places (the fence, connect, and disconnect contribution paths; the two
late-registration re-checks in ``pmix_server.c``; and the lost-connection
handler). All of them must consult one shared predicate so the accounting
can never diverge between call sites.

Lost-connection accounting rules
---------------------------------

When a connection is lost, for every tracker (on **both** the
``collectives`` and ``grp_collectives`` lists) in which the departing
peer is an expected participant, the server must apply exactly one of the
following, based on whether that specific participant has already
contributed:

**Case A — the participant had already contributed.**
   *Ignore the loss for purposes of this collective.* The contribution
   stands: it remains in ``contributed`` and its delivered data remains
   part of the assembled collective. The expected count is **not**
   reduced and the contribution is **not** removed. The participant has
   already discharged its obligation to this collective; its subsequent
   death is irrelevant to local completion.

**Case B — the participant had not yet contributed.**
   The participant can never contribute now, so move it from
   ``expected`` accounting into ``departed`` (in the counter view: it is
   removed from the set of contributions still being waited on). This
   allows the collective to complete once the remaining live participants
   contribute, rather than hanging forever on a participant that will
   never call.

In both cases the completion predicate is then re-evaluated. If the
collective has become locally complete as a result, it is processed
exactly as it would be on a normal final contribution (see *Interaction
with the host handoff* below).

Note the asymmetry with the current implementation, which in every case
both decrements the expected count *and* removes any prior contribution.
Case A above forbids both of those adjustments; that is the specific
change that fixes the vulnerability.

Status reporting on loss
~~~~~~~~~~~~~~~~~~~~~~~~~

When a loss affects a collective, the server continues to report the
outcome to the surviving participants via the
``PMIX_LOCAL_COLLECTIVE_STATUS`` info field on the tracker:

* If at least one expected participant still remains after the loss is
  accounted for, the status is ``PMIX_ERR_PARTIAL_SUCCESS``.
* If the loss leaves no remaining live participants, the status is
  ``PMIX_ERR_LOST_CONNECTION``.

A loss falling under Case A (the participant already contributed) does
not by itself change the set of remaining participants and therefore does
not, on its own, downgrade the status.

Required architecture: a single tracking service
-------------------------------------------------

The behaviors above must be provided by **one collective-tracking service
with a single code path**, shared by all four operations, rather than by
per-operation copies of the accounting logic.

Today the logic is duplicated. Fence, connect, and disconnect use
``pmix_server_trkr_t`` on ``pmix_server_globals.collectives``; group
operations use the parallel ``grp_block_t`` / ``grp_trk_t`` pair on
``pmix_server_globals.grp_collectives``. The two carry essentially the
same state — participant list, expected count, definition-complete flag,
contribution list, host-handoff flag, lock, event — split across slightly
different layouts, and the completion test is hand-rolled at each of the
six-plus call sites that ask "is this collective locally complete?" This
duplication is precisely what let the group path drift out of sync: the
lost-connection handler was never taught about ``grp_collectives``, so
group collectives receive no loss accounting at all.

Consolidating removes that class of defect by construction. The service
must own:

* tracker lifecycle (locate-or-create, release);
* the identity accounting — the ``expected`` / ``contributed`` /
  ``departed`` sets;
* the single completion predicate defined above, evaluated in exactly one
  function;
* lost-connection accounting (Case A / Case B), applied uniformly to
  every tracker on a single tracker list;
* the host-handoff gate (``host_called``); and
* delivery of results to the surviving local participants.

Behavior that genuinely differs between operations must be confined to a
small set of per-operation hooks the service invokes — which host
callback to forward to, and how to assemble the operation's payload
(modex-data collection for fence; context-id assignment and group-info
aggregation for group construct). Everything else is shared. Sharing the
logic in exactly one place is the only tractable way to keep the
invariants below true: a predicate or a rule that exists once cannot
drift out of agreement with a copy of itself.

**What "single" means in practice.** The fence, connect, and disconnect
operations do share one tracker type (``pmix_server_trkr_t``), and for
them the requirement is literal: one struct, one predicate, one loss
routine. The group operations are the exception. Their tracker is a
two-level structure — a ``grp_block_t`` owning a list of ``grp_trk_t``
members, one per call signature — because a single group construct can
be assembled from several distinct signatures (leader, followers,
bootstrap), the contribution count is the number of member trackers
rather than the size of a single contribution list, and the block carries
group-only state (context-id assignment, group-info aggregation). Forcing
that onto ``pmix_server_trkr_t`` would be an invasive rewrite of delicate,
HPC-critical code that currently has no automated test coverage. The
consolidation requirement for the group family is therefore satisfied by
sharing the *semantics* — the same completion-predicate shape
(contributed + departed ≥ expected) and the same Case A / Case B
lost-connection rules — expressed in the group's own structure and
reached through a single exported entry point, rather than by collapsing
the two structs into one. Invariant 7 below is stated in these terms.

Additional required behaviors
-----------------------------

**Clones share a rank but must not corrupt the accounting.** A process
may fork/exec one or more clones that connect to the same server with the
same ``{namespace, rank}`` and independently join a collective. The
overriding requirement is that losing one peer must never discard a
clone's contribution that shares the same name. The current expected
count (``nlocal``) is maintained per *rank* — a clone is not counted as a
separate expected participant — so the contribution question is asked per
rank too: a rank is satisfied once any of its peers has contributed. The
clone data-loss hazard is closed not by keying the contribution *check*
on the peer pointer but by the stronger guarantee that **a loss never
removes a contribution at all** (Case A does nothing): whatever a rank's
peers delivered stays in the collective regardless of which peer's
connection drops. (A future change that made ``nlocal`` count clones
individually would move the contribution check to the peer granularity;
until then, per-rank name matching is the consistent choice.)

**Contribution is idempotent and terminal.** A given required
contribution is recorded once. Once recorded, it is never removed by
loss accounting (Case A). A duplicate call for the same required
contribution must not double-count toward completion.

**Late registration must not race the accounting.** Namespaces and
clients may register after a collective has already begun (the host may
spawn a client that calls in before the local ``register_client`` event
fires). The paths that raise ``nlocal`` / grow ``expected`` on late
registration and then re-test completion must use the same identity-based
predicate, so that a late registration and a concurrent loss cannot
combine to complete a collective early.

**Interaction with the host handoff.** Once a tracker has been forwarded
to the host (``host_called`` is set), local completion has already
occurred and the local accounting is frozen: a subsequent loss must not
attempt to re-complete or re-forward the collective. The server waits for
the host to return, then delivers the result to whichever local
participants are still connected. Losing a participant after handoff only
affects whether a reply can be delivered to that participant — never the
completion or contents of the collective.

**Group operations have the same requirement.** The group
construct/destruct trackers (``grp_block_t`` / ``grp_trk_t`` on
``grp_collectives``) use the same counter-based accounting and are
subject to the same vulnerability. They must adopt the identity-based
model and must be traversed by the lost-connection handler, which today
walks only the ``collectives`` list.

**A voluntary group leave is the deliberate cousin of a loss.** When a
member calls ``PMIx_Group_leave`` it will never contribute to an
in-flight construct/destruct of that group, exactly as a lost member
never will — the only difference is that the departure is intentional and
scoped to one named group rather than triggered by a dropped socket. The
group family therefore accounts a voluntary leave through the *same*
``departed`` machinery as a loss: the per-block routine
(``account_departed``) is reached from two entry points —
``pmix_server_grp_peer_lost`` (walks every block for a dropped peer) and
``pmix_server_grp_member_left`` (walks only the blocks for the named
group, invoked when the server intercepts the ``PMIX_GROUP_LEFT`` event
the leaving client generates). Case A / Case B and the completion
predicate are identical; a leave that completes a block's local phase
forwards it to the host on the survivors just as a loss does.

Invariants
----------

An implementation of this specification must preserve the following at
all times:

#. **A contribution, once recorded, is only ever removed by delivering
   the collective's result to that participant — never by loss
   accounting.**
#. **The expected participation set is reduced for a lost participant
   only if that participant had not yet contributed.**
#. **Local completion is decided by one shared predicate**, evaluated
   identically at every call site, expressed in terms of the
   ``expected``, ``contributed``, and ``departed`` sets.
#. **A collective completes locally if and only if its definition is
   complete and every expected participant has either contributed or
   departed-before-contributing.**
#. **After ``host_called`` is set, local loss accounting never re-drives
   completion** for that tracker.
#. **The rules apply uniformly** to fence, connect, disconnect, and group
   operations, and to both directly-connected clients and their clones.
#. **Participation logic is shared, not copied.** Fence, connect, and
   disconnect are tracked by one tracker representation with one
   completion predicate and one lost-connection routine. Group operations,
   whose two-level tracker differs materially, reuse the same completion-
   predicate shape and the same Case A / Case B loss semantics through a
   single exported entry point rather than a private copy. No operation
   carries its own divergent copy of the participation rules.

Relationship to the current implementation
-------------------------------------------

The behaviors above are partially and fragilely approximated today: the
fence/connect/disconnect path attempts to remove a departed peer's
contribution from ``local_cbs`` while also decrementing ``nlocal``, but
it does so by process name (so it cannot distinguish clones), it discards
already-delivered data (violating invariant 1), it duplicates the
completion test across many call sites, and it does not cover group
operations at all. The implementation plan describes how to collapse the
two tracker families into the single identity-based tracking service
required here — one tracker representation, one completion predicate, one
lost-connection routine — while preserving the established tracker
lifecycle, the thread-safety (progress-thread / caddy) rules, and full
ABI and wire-format compatibility (the tracker structures are internal to
``libpmix``; no public type, API signature, or on-wire message layout
changes).
