Deferred work
=============

Work that is knowingly incomplete, with enough of a pointer to pick it up.

This is **not** the issue tracker: anything a user has reported belongs at
https://github.com/openpmix/prrte/issues.  What is collected here is the
other kind — a path the code takes deliberately today because the better one
was left for later, a test that cannot exist yet, a measurement nobody has
been able to take.  Each entry says where the evidence is, so that the next
person can decide whether it is still true before acting on it.

Add to it when you leave something undone, and delete from it when you do
the work.  *Every marker* left in PRRTE's own sources — a ``TODO``, a
``FIXME``, an ``XXX``, an arm deliberately left empty — belongs here in
prose; **Every marker in the code** is the table that makes that
correspondence checkable, and a marker missing from it means one of the two
is stale.

The two sections after that table are of a different kind.  **Review status**
records how far the file-by-file review pass has reached, and — the part that
matters more — which subsystems have been rebuilt since their review, so a
guide's authority can be judged before it is trusted.  **Decided against** is
the other outcome: work that was investigated and deliberately not done, kept
so nobody spends the effort a second time.

Runtime behavior
----------------

**``ras/flux`` has no ``modify()``.**  It returns ``PMIX_ERR_NOT_SUPPORTED``
(``src/mca/ras/flux/ras_flux_module.c``), so the elastic extend/release
surface exists for SLURM only.  Everything above the component is
RM-agnostic; what is missing is the Flux-side conversation.

**Mixed allocators are not supported, and would need more than the ``ras``
framework.**  The motivating case is a cloud/local combination: an allocation
from a scheduler plus a set of unmanaged nodes outside it.  The ``ras``
framework was made multi-select for this, and it never delivered it -
``prte_ras_base_allocate`` breaks at the first module that succeeds, so there
is no union step and the second allocator never contributed a node.  That
selection is now single, which loses nothing that worked.  Supporting the case
properly is a larger piece of work than re-admitting multiple ``ras`` modules:
``plm`` is not multi-select either, so nothing tracks which launcher owns
which nodes, and the daemon-launch path would have to fan out through more
than one.  At minimum it needs a launcher affinity recorded per node and
honored by ``prte_plm_base_setup_virtual_machine``.

**The bootstrap configuration parses one option it deliberately does not
publish.**  ``SessionTmpDir`` is read into the bootstrap configuration and
then left there (``prte_ess_base_bootstrap_params``,
``src/mca/ess/base/ess_base_bootstrap.c``): the facility it would drive — a
session-directory base for application jobs distinct from the DVM's own —
does not exist, so publishing it as an MCA envar would promise behavior the
daemon does not have.  The parse stays because the file format is the
specification; the plumbing waits on the facility.

What is missing is small but not free.  ``_setup_job_session_dir``
(``src/util/session_dir.c``) roots every job directory at
``prte_process_info.top_session_dir`` unconditionally, and no MCA parameter
reroots it — ``prte_tmpdir_base`` and its local/remote siblings move the
*whole* tree.  It is worth having: ``jdata->session_dir`` is what PRRTE hands
PMIx as ``PMIX_NSDIR`` (``pmix_server_register_fns.c``), which is where the
gds/shmem segments and an MPI's shared-memory backing files land, so
separating that from the DVM's own small control directory is a real
operational knob.  Two things have to be decided before writing it:

* **Uniqueness.**  Today it comes from the ``<prefix>.<host>.<pid>`` level
  above the job directories.  A bare ``<SessionTmpDir>/<jobid>`` collides
  between two DVMs on one node, so that level has to be replicated under the
  new root — and ``prte_job_session_dir_finalize`` then has two roots to
  destroy rather than one.
* **What the clients are told.**  ``PMIX_SERVER_TMPDIR`` must stay the
  daemon's own directory (it is where client rendezvous files go), so
  ``PMIX_NSDIR`` would no longer sit beneath it.
  ``docs/how-things-work/session_dirs.rst`` says "usually placed underneath",
  not "must be", so this is legal — but it is a change a client can observe.

The ``Log*`` half of this entry is **closed, not implemented**: the six
``ControllerLog*`` / ``PRTEDLog*`` keys have been removed from the file
format rather than plumbed.  The facility they would have driven now exists
as the ``state_base_log_jobstate`` / ``state_base_log_procstate`` /
``state_base_log_path`` MCA parameters
(``src/mca/state/base/state_base_log.c``), and that is the only way to ask
for it.  A configuration key applies to every daemon of every DVM the
cluster starts, for as long as the line is present and with nobody watching,
while the volume of a per-transition record scales with the processes
launched and lands on the same disk the session directories use — a knob
with that blast radius has to be asked for by the run that wants it.  An
older ``prte.conf`` carrying the keys still parses: they are unknown keys
now, and unknown keys are ignored.

**A fence cannot tell a post-release straggler from the next round.**  A
fence signature is only its participant list, so nothing on the
``PRTE_RML_TAG_FENCE`` wire says which *round* a contribution belongs to.
That is invisible in the normal flow — a daemon converges only once every
contribution it expects has arrived, so nothing can arrive afterwards — but
the controller also ends a fence early, on a ``PMIX_TIMEOUT`` and on a
participant lost to a failed daemon (``abort_fence_op()``,
``src/grpcomm/grpcomm_fence.c``).  A contribution still climbing the tree
then lands on a daemon that has already retired its tracker, and
``fence_recv()`` builds a fresh one for it.  The next fence over the same
participants finds that tracker, inherits its ``nreported`` and its bucket,
and can converge early carrying the previous round's data.

The group collective solves the same problem with ``completed_group_ops``, a
bounded memo of released operations that ``grp_recv`` consults, and that fix
**cannot** be copied here.  A group is keyed by ``groupID`` plus operation
and ``group()`` forgets the memo entry when a local client starts one; a
daemon relaying a fence for its subtree has no local client, so the memo
would never be forgotten there and the *next* fence's legitimate
contribution would be dropped — a hang in place of a wrong answer.  On a
pure relay the two are genuinely indistinguishable without a round
discriminator on the wire, and a fence has no originator to assign one: this
is the same "every participant must reach the same answer independently"
problem that the withdrawn lateral fence ran into
(``src/grpcomm/AGENTS.md``).

What would work is a per-signature release count: each daemon counts the
releases it has seen for a signature, stamps its contributions with that
count, and drops a contribution stamped below its own — the recovery epoch's
mechanism, scoped to a signature and driven by releases rather than by
failures.  That is a wire change plus a memo that holds a counter, and it
needs the dockerswarm harness to validate, so it has not been attempted.

**A job cannot ask for a transport.**  The network allocation request the
odls builds for each job names ``<nspace>.net`` and a security key and
otherwise takes whatever transport the resource manager offers by default
(``prte_odls_base_default_get_add_procs_data``,
``src/mca/odls/base/odls_base_default_fns.c``).  There is no command-line
surface for saying which one, and the note there is the record that one was
always intended.

Test coverage
-------------

**Init and finalize.**  Nothing covers the ``prte_init``/``prte_finalize``
sequence itself beyond a smoke test, and nothing configures with
``--disable-per-user-config-files``, so the ``#else`` arm of
``PRTE_WANT_HOME_CONFIG_FILES`` is never compiled in CI
(``src/runtime/AGENTS.md``).

**``--disable-pretty-print-stacktrace``.**  Nothing in CI configures with it,
which is how the arm came to not build at all: with the feature off, two
statics in ``src/util/stacktrace.c`` were unreferenced and ``--enable-debug``
made that an error.  That is fixed, but the option is still compiled only
when somebody thinks to try it, and the ``contrib/platform`` files that set
it are not built either.  It is one more configuration in the same class as
the ``#else`` arm above: cheap to break, and nothing watching.

**Heterogeneous DVMs.**  The byte-order helpers (``prte_hton64`` /
``prte_ntoh64``) are exercised by every inter-daemon message, but the case
they exist for — two daemons that disagree about endianness — cannot be built
from a container swarm on one host.  No test can currently fail if they are
wrong.

**The resource managers nobody has.**  ``ras``/``plm``/``ess`` components for
PBS, LSF, gridengine, Flux and PALS are compile-only in CI
(``--enable-testbuild-launchers``, and for the ones needing third-party
headers, against declaration-only stubs).  A build proves they still compile;
only a real allocation on such a system proves anything else.  SLURM is the
exception — ``contrib/slurmswarm`` runs a real one.

**macOS.**  ``contrib/dockerswarm/run-tests.sh macos`` is a single-host
subset by construction.  Everything multi-node on that platform is untested.

**``--enable-mca-dso`` beyond one node.**  Building every component as a
run-time loadable DSO instead of linking it into ``libprrte`` is now built,
unit-tested and smoke-launched on Linux by the ``ubuntuMcaDso`` job in
``.github/workflows/builds.yaml``, and the same build was verified by hand
on macOS (26 components, ``make check`` clean, ``prterun -n 2 hostname``).
That is a single node, so it proves the components load and that the ones a
local launch needs work; it does not exercise a component only reached
across daemons.  ``contrib/dockerswarm`` can now be built that way —
``PRTE_SWARM_MCA_DSO=1 ./build.sh``, after which the ordinary suite is the
test — but the arm is **opt-in**, so what is still missing is anyone
running it as a matter of course.  Note that the CI job asserts on a
*named* component being loadable rather than on a count: the default
build already installs a DSO or two of its own, so a count cannot tell
the two configurations apart, and a silent fallback to static linking
would otherwise leave every step passing while testing nothing.

**``SLURM_TASKS_PER_NODE`` in its single-node spelling.**  The suite asserts
the ``2(xN)`` form that a multi-node allocation produces; the ``2(x1)`` form
goes through the same parser and is not separately covered.

**A revived daemon, and RELM's depth-stamped link updates.**  The unheal
path (``docs/plans/bootstrap/unheal_plan.rst``) is implemented and
harness-verified for the routing recompute, for the xcast op-id stream a
late joiner rejoins, and for the nidmap span the handoff encodes.  One item
in it is argued rather than tested: RELM stamps link updates with the tree
depth and ``update_link`` drops one whose depth does not match, while a
revival changes depths and rides the xcast forward-first.  The static
argument is that every daemon recomputes synchronously right after
forwarding, so both ends have settled before any link update — a later,
separate message — is processed.  The case that would settle it is cheap:
kill an interior node on the Docker harness, restart it, then launch a job
across the whole DVM and check that nothing was lost.

**The topology sensing path, and a cpuset wide enough to fill the buffer.**
``prte_hwloc_base_get_topology()``'s sensing arm reads the real machine, so
nothing drives it — every unit test and the offline harness hand it a
topology file instead.  Nor is ``prte_hwloc_print()`` exercised against a
machine of a few thousand PUs, which is the width its cpuset buffer is sized
for (``src/hwloc/AGENTS.md``).

**The XML arm, and the copy constructors.**  In
``src/runtime/data_type_support/``, the print functions' XML output has no
test, and ``prte_job_copy`` / ``prte_proc_copy`` have no callers to be tested
through.

Every marker in the code
------------------------

The list above is meant to be the whole of it: a ``TODO``, ``FIXME`` or
``XXX`` left in PRRTE's own sources should have an entry here in prose — and
so should an arm deliberately left empty, which carries no keyword at all —
and this table is what makes that checkable.  If you leave a marker, add the
entry; if you find a marker this table does not name, either the entry or
the marker is stale.

.. list-table::
   :header-rows: 1
   :widths: 45 55

   * - marker
     - entry above
   * - ``mca/odls/base/odls_base_default_fns.c``,
       ``prte_odls_base_default_get_add_procs_data``
     - a job cannot ask for a transport
   * - ``mca/ess/base/ess_base_bootstrap.c``,
       ``prte_ess_base_bootstrap_params``
     - one bootstrap option is parsed and not plumbed
   * - ``mca/ras/flux/ras_flux_module.c``, ``modify``
     - ``ras/flux`` has no ``modify()``

Two families of marker are **not** ours, and are deliberately absent from
that table: the ``TODO`` comments in ``hostfile_lex.c`` and
``rmaps_rank_file_lex.c``, which come from flex's generated skeleton, and the
``FIXME`` comments throughout ``config/libtool.m4`` and ``config/ltmain.sh``,
which are vendored Autotools.

Review status
-------------

PRRTE's source has been going through a file-by-file review pass: read the
code, verify each finding adversarially, fix what survives, and write what
the review established into the directory's ``AGENTS.md``.  This section
records where that pass has reached, because "has anyone actually looked at
this?" should be a question with an answer, and because a review is only as
good as the code it was a review *of* — several subsystems have been rebuilt
since theirs.

The churn figures are commits and diff lines touching the subsystem's
sources since its own review commits, measured on **2026-08-17**.  They age;
regenerate them with ``git log --since=<date> -- <dir>`` rather than trusting
them a month from now.  ``AGENTS.md`` and ``CLAUDE.md`` are excluded from the
counts, since a review's own write-up would otherwise count as change since
the review.

Reviewed, and stable since
~~~~~~~~~~~~~~~~~~~~~~~~~~

What each of these directories' ``AGENTS.md`` says is still what the code
does.

.. list-table::
   :header-rows: 1
   :widths: 26 24 50

   * - subsystem
     - last reviewed
     - since
   * - ``src/event``
     - 2026-07-31
     - nothing
   * - ``src/include``
     - 2026-08-02 (round 2)
     - nothing
   * - ``src/pmix``
     - 2026-07-30
     - 6 commits, ~110 lines
   * - ``src/mca/filem``
     - 2026-08-03 (round 2)
     - 3 commits, ~30 lines
   * - ``src/mca/ess``
     - 2026-08-03 (round 2)
     - 4 commits, ~120 lines
   * - ``src/mca/iof``
     - 2026-08-04 (round 2)
     - 6 commits, ~150 lines
   * - ``src/tools``
     - 2026-07-29
     - 11 commits, ~120 lines

Reviewed, but changed materially since
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These carry a review, and then the ground moved.  The review's findings are
still worth reading — they say what the invariants are — but nothing here has
been re-verified against the code as it now stands, and the top of the list
is not a marginal case.

.. list-table::
   :header-rows: 1
   :widths: 22 20 58

   * - subsystem
     - last reviewed
     - since
   * - ``src/grpcomm``
     - 2026-08-02 (round 3)
     - The subsystem was then taken out of MCA and rebuilt at a new path
       (``src/mca/grpcomm`` became ``src/grpcomm``), collective movements
       were added and most of them removed again, and the xcast was
       reworked twice.  Every line is now at an address the review never
       saw.  **Re-review this one first.**
   * - ``src/prted``
     - 2026-07-30
     - 43 commits, +3,187/-482.  The PMIx server host module
       (``src/prted/pmix``) absorbed most of the launch-message work, the
       lazy proc-data derivation, and the tool-connection data server.
   * - ``src/mca/ras``
     - 2026-07-26
     - 44 commits, +2,093/-494.  Elastic extend/release, the SLURM
       ``--json`` parser and its version gate, node reservation.
   * - ``src/mca/rmaps``
     - 2026-07-28
     - 28 commits, +2,408/-480.  Per-device mapping is nearly all of it,
       including a new enumerator vtable in the base.
   * - ``src/runtime``
     - 2026-07-29
     - 30 commits, +1,792/-635.  Placement packed as maps, the cpuset
       scatter, the mode byte at the head of the job buffer, and the data
       server's move onto a PMIx tool connection.
   * - ``src/mca/schizo``
     - 2026-07-28
     - 30 commits, +1,535/-522.  Device-mapping qualifiers and the
       debugger options.
   * - ``src/rml``
     - 2026-07-29
     - 20 commits, +1,522/-190.  The OOB wire header lost its namespaces,
       per-peer worker bases arrived, and RELM gained demotion handling
       (RELM was reviewed as part of this pass, not separately).
   * - ``src/util``
     - 2026-07-29
     - 22 commits, +889/-96
   * - ``src/mca/odls``
     - 2026-08-04 (round 2)
     - 8 commits, +543/-289.  The receiving half of the cpuset scatter.
   * - ``src/hwloc``
     - 2026-08-01 (round 3)
     - 11 commits, +536/-437.  Device enumeration and the UUID plumbing.
   * - ``src/mca/state``
     - 2026-07-29
     - 17 commits, +424/-120
   * - ``src/mca/errmgr``
     - 2026-08-02 (round 2)
     - 15 commits, +301/-217
   * - ``src/mca/plm``
     - 2026-08-04 (round 2)
     - 13 commits, +279/-147

Not yet reviewed
~~~~~~~~~~~~~~~~

Every directory below has an ``AGENTS.md`` — the orientation guides were
written across the tree in one pass — but an orientation guide is a
description, not a review.  Nobody has read these for defects.

**``src/mca/common``** (with ``common/slurm``).  Written 2026-08-06 and
untouched by any review.  It is now the single place every SLURM component
asks "is this really SLURM", so a wrong answer there is wrong in ``ras``,
``plm`` and ``ess`` at once.

**``src/mca/prtereachable``**, **``src/mca/prtebacktrace``**,
**``src/mca/prteinstalldirs``**.  Nothing but a header-installation fix and
the framework-version stamp has touched these since the pass began.  They are
small and quiet, which is the argument both for leaving them and for the fact
that a defect in them would have gone unnoticed.

**``test/unit`` and the harness scripts under ``contrib/``.**  Reviewed
informally as they were written; never subjected to the pass.  This is worth
saying out loud, because the harness is what decides whether everything else
passes.

Performance work: what the measurements settled
-----------------------------------------------

Nothing in this section is outstanding work.  What is here is the evidence
behind the message-size changes that have landed, and the traps that made
each of them expensive to get to — kept so the next person can extend the
work without re-deriving the ground, or re-proposing something already
measured and answered.

**Judge a header change on what a small message costs.**  ``prte_oob_tcp_hdr_t``
rides every RML message, and it used to carry the origin and the destination
as two ``pmix_proc_t`` — 552 bytes, of which 512 were two fixed 256-byte
nspace arrays holding the same short string.  On small messages that was most
of the traffic: a launch-message cpuset slice for an eight-process node is
101 bytes of payload, so the header was 85% of it.  It is now **30 bytes,
fixed**: two ranks and no namespace at all.

The namespace came out in two steps, and the second one is the useful lesson.
Collapsing the two ``pmix_proc_t`` to two ranks plus one length-prefixed
nspace took it to ~50 bytes and was easy, because the two names had never
differed.  Removing the last ~20 was not a matter of trimming further: what
stood in the way was that "both ends are always this daemon's namespace" was
an *invariant nobody enforced*, and the failure mode if it were ever broken is
a message delivered under the wrong sender identity rather than an error.
Two things settled that.  A **data server hosted by another DVM**
(``prte_pmix_server_uri``, historically ``ompi-server``) was the one feature
that ever wanted to address a foreign namespace, and it turned out never to
have worked over the RML — every send entry point has taken a rank in the
sender's own namespace since the 2022 rework, so the server's namespace was
silently discarded — so it now crosses on a PMIx **tool** connection instead
(:doc:`plans/cross_dvm_data_server/cross-dvm-data-server`).  And the invariant
became a **check**: only daemons open an OOB endpoint, the connect handshake
still carries a namespace, and ``tcp_peer_recv_connect_ack`` refuses a peer
whose namespace is not ours.  Verified once per connection is what lets every
message omit the field.

**Judge this kind of change on wire bytes, not on the raw message size.**
A launch-message size is usually quoted raw, which is the right unit for "how
much buffer does the master build and deflate" and the wrong one for "how
much crosses the network": the xcast compresses before it forwards, and a
field holding the same value in every record can compress away to nothing
while looking enormous raw.  Measured on a live 32-node broadcast at 8 ppn
(``grpcomm_base_verbose 1``, tag 18 is the launch message):

=====================  ===========  ============  =====
what                   raw          wire          ratio
=====================  ===========  ============  =====
whole message          3,790 B      1,280 B       0.34
per proc (slope)       13.0 B       --            --
cpuset (core vs none)  2.0 B/proc   0.77 B/proc   0.40
=====================  ===========  ============  =====

**The cpuset does not compress away, and more repetition does not help it.**
Its marginal wire cost held flat — 0.766, 0.742, 0.797 B/proc — across 8, 16
and 32 nodes, i.e. a fourfold increase in how often the same handful of
cpuset strings recur.  The overall ratio improves with scale (0.42 -> 0.34)
only because the fixed job-level part amortizes.  So ~0.4 is the marginal
ratio to extrapolate with, and it is what said the scatter was worth
building before it was built.

**Do not model this instead of measuring it.**  A synthetic model of the
per-proc region — the right fields, the right varint encoding, a faithful
cpuset distribution — predicted the region would deflate ~180:1 and the
cpuset would cost 0.05 B/proc on the wire.  The real answer is 0.77, a factor
of fifteen out, and the conclusion drawn from the model ("the compressor
already solves this, drop the idea") was the opposite of the one the
measurement supports.  The model is not salvageable by refining it; the
per-proc region does not exist in isolation on the wire.

**Two things settled along the way, so they need not be re-derived:**

* *The state is not worth scattering.*  Nothing in the tree ever sets a proc
  state to ``PRTE_PROC_STATE_RESTART`` — the only occurrence is
  ``prte_pmix_convert_pstate()`` translating an incoming PMIx state — and
  nothing sets ``PRTE_JOB_FLAG_RESTARTED`` either.  Both consumers
  (``odls_base_default_fns.c`` and ``filem_raw_module.c``) treat ``INIT`` and
  ``RESTART`` identically, so the wire never needs to tell them apart.  It is
  ~2 B/proc raw and varint-squashed, and it genuinely varies for the *job
  catchup* caller, which packs already-running jobs whose proc states feed
  ``PMIX_QUERY_PROC_TABLE``.  Leave it alone.  The **node rank** is likewise
  staying in the broadcast: a remote node-rank get that falls through to
  PMIx's one-job-per-node assumption returns a *wrong* value rather than
  nothing.
* *An optional trailing field per proc does not work.*  ``prte_proc_pack``
  runs in a loop and more job fields follow the array, so an absent flag is
  ambiguous both against the next proc's first field and against
  ``stdin_target``; ``PMIX_ERR_UNPACK_PAST_END`` never fires because there is
  always more buffer.  Any conditional field needs a discriminator the
  decoder has already read — which is why the shape is now declared by a
  mode byte at the head of the buffer.

**Measuring the launch message at all takes three settings**, and each one
silently gives a wrong answer if omitted: ``--prtemca hwloc_use_topo_file``
with a NUMA-bearing topology (a dev box and the swarm containers have no NUMA
node, so the default ``NUMA:IF-SUPPORTED`` binds nothing and you measure the
unbound shape while believing otherwise); a PMIx built *without* debug (see
below); and, for a wire number, a payload above the compressor's
``pcompress_base_limit`` — below it ``wire`` simply equals ``raw`` and the
census tells you nothing about the ratio.

**Small effects cannot be measured end-to-end on the container swarm.**  At
40 nodes the collect fence's wall clock has a run-to-run coefficient of
variation of 35–50%, so anything under about a third is not resolvable there
however many arms the sweep has — an attempt to settle whether compressing
the xcast payload helps (an effect of 1–7%) produced overlapping ranges whose
*sign* flipped between radices.  Measure the mechanism directly instead
(``--prtemca grpcomm_base_verbose 1`` reports the size, ratio and
microseconds of every broadcast).  See ``contrib/dockerswarm/AGENTS.md``,
§18.

Decided against
---------------

Work that was looked into and deliberately not done.  These entries are here
so that the next person does not rediscover the idea and spend the effort
again: each says what was measured and what the measurement decided.  Moving
one back up the page takes new evidence, not a fresh reading of the same
facts.

**Guarding the stack-trace handler against a platform with no
``siginfo_t``.**  ``show_stackframe`` (``src/util/stacktrace.c``) is
installed as an ``sa_sigaction`` handler and reads ``si_code``, ``si_addr``
and their neighbors unconditionally.  A ``FIXME`` carried over from the
code's Open MPI ancestry asked for a second, plain ``sa_handler`` arm for
"systems which don't have siginfo".  There are none.  ``siginfo_t``, the
``sa_sigaction`` member of ``struct sigaction`` and the ``SA_SIGINFO`` flag
are all mandatory in POSIX.1-2001 — not one of its option groups — and
PRRTE's floor sits far above that: C11, PMIx 7, hwloc 2.1, libevent 2.0, and
unguarded POSIX-2001 calls in every direction.  The comment dates from 2004,
when the Unixes it was written for were still in the field.

What the probes around it check is the part that genuinely is optional, and
that division is the right one: ``si_fd`` is not in POSIX at all (a
Solaris/Linux extension — macOS does not have it, as any build tree's
``prte_config.h`` shows), and ``si_band`` belongs to the obsolescent XSI
SIGPOLL option.  Both are probed, and both are guarded at their single use;
so is every ``si_code`` value the switch decodes, and so is ``SIGPOLL``
itself.  The only thing taken on faith is the structure.

A second handler would therefore be code no compiler anywhere would ever be
handed, living in a signal handler, where a mistake is untraceable.  Nor is
a ``configure`` probe for the structure worth having: a check that can only
ever answer yes is the same dead weight in a different file, and guessing at
what a platform that has never existed would want is not portability.  The
platform PRRTE cannot serve here already has its answer —
``--disable-pretty-print-stacktrace`` compiles the handler out entirely and
is used by several ``contrib/platform`` files.

Checking that claim was worth more than the probe would have been: the arm
did not build.  ``unable_to_print_msg`` and ``set_stacktrace_filename`` sat
outside the switch their only users live behind, so with the feature off
they were unreferenced and ``--enable-debug`` rejected the file.  Both now
live inside the switch, and the disabled build compiles warning-free, passes
``make check``, and launches.  That nobody had noticed is its own entry
under **Test coverage**.

So the ``FIXME`` is gone and nothing replaced it but a comment saying what is
assumed and why.  If a platform without ``siginfo_t`` ever turns up, it will
say so as a compile error naming the type in this one file, and the
disable option is right there.

**Giving RELM an MCA variable to choose between reliability modules.**  The
marker in ``prte_relm_register`` asked for the enum variable that would select
one, and the entry that carried it said the work was only worth doing if a
second module was.  None was: RELM shipped with a ``prte_relm_module_t`` of
four function pointers that only ever held the base module's, a ``base/``
subdirectory holding the only implementation, and a second bundle of nine
callbacks on the state machine that the base module's ``init()`` wired to its
own functions.  Two dispatch surfaces, one implementation, and nothing that
could pick anything.

The one candidate second module was a passthrough that degraded a reliable
send to an ordinary one — an escape hatch and an A/B lever, since the base
protocol walks the path three times per message (data down, ACK up, ACK-ACK
down) and rides paths as busy as every IOF chunk from a daemon to the HNP.
That is an on/off switch, not a choice between strategies, and it would hand a
user a way to silently lose the messages this layer exists not to lose.  A
genuinely different strategy — end-to-end acknowledgement, journaled delivery
— has no requirement behind it and no author.

So the indirection was removed rather than completed, the way ``routed`` and
``oob`` were folded into ``src/rml`` and ``grpcomm`` was de-framework'd before
it: ``base/`` is flattened into ``src/rml/relm/``, the protocol's entry points
are ordinary functions the engine calls directly, and ``relm.h`` is four
symbols wide.  The split that survives is structural — ``state_machine.c`` is
the generic engine, ``state_updates.c`` and ``link_updates.c`` are the
protocol on top of it — and it is a compile-time boundary rather than a
dispatch one.  Reintroducing dispatch is cheap against an interface that
narrow, so it can be done when a second implementation actually exists.

**Making the routing-tree repair the one place that raises
``COMM_FAILED``.**  ``prte_rml_repair_routing_tree`` ends by calling every
subsystem's ``fault_handler`` in turn, and a marker there asked whether that
one place should also set the departed ranks to
``PRTE_PROC_STATE_COMM_FAILED`` rather than leaving each caller to do it.  It
should not.  Repairing the tree and telling the errmgr a daemon is gone are
different questions, and the code has both halves without the other:

* A DVM shrink repairs in one batch for the whole campaign and must **not**
  raise it — ``shrink_campaign_complete`` (``src/mca/ras/base/ras_base_allocate.c``)
  tears the targets out of the DVM itself, and the already-departed guard in
  ``errmgr/dvm`` exists precisely to swallow the comm-failure their real
  departure produces afterwards.
* During an ordered teardown ``prte_rml_route_lost`` returns without repairing
  anything, yet the OOB still raises ``COMM_FAILED`` — and that raise is what
  drives "all my routes and children are gone, terminate" in *both* errmgr
  components.  Centralizing the raise would break orderly shutdown.
* ``errmgr/prted`` heals the tree around a peer it has given up connecting to
  from inside its own error handler, where raising the state again would
  re-enter it.

Four of the six call sites therefore repair without reporting, for three
different reasons, and one report happens where no repair does.  What the
paths that *do* report must not do is diverge, because the routing tree's
``failed_dmns`` set is what they all use to tell a first report from a
duplicate — a path that repairs without reporting makes every later report
for that rank look like a duplicate and suppresses it for good.  They are
collected behind ``report_new_departures`` in
``src/rml/rml_fault_handler.c``; the two lineage-inference paths used to be
outside it, and are not any more.

**Aggregating the launch-message cpuset slices per next hop.**  The scatter
itself has landed: the launch message is packed ``PRTE_JOB_PACK_NO_CPUSETS``
and each daemon is sent the bindings of the procs it will fork, point to
point (``prte_odls_base_send_cpuset_slices``).  Measured with ``--rtos
donotlaunch`` at 1000 x 128 on a 176-core, 5-NUMA topology, that takes the
raw launch message from **1,622,647 to 602,647 bytes** — 8 B/proc, 63% of
what was left after the maps and the empty attribute list came out.

What was proposed on top of that was how those slices leave the master.  It
is one routed send per daemon: the *bytes* aggregate, because
``prte_rml_get_route`` sends each toward its target and the tree carries a
subtree's slices over one link, but the *messages* do not — a 1000-node job
posts 999 sends from the master on the launch path.  One message per next
hop, each carrying the slices for that child's whole subtree and split by the
relay, would make that O(radix).

Measured, that buys very little.  The payload is unchanged either way: a
slice crosses exactly the links between the master and its daemon whichever
way it is bundled.  What aggregation removes is one *message header* per
crossing it saves, and at the default radix of 64 the crossings (Σ depth over
the daemons) fall from 1,934 to 999 at a thousand nodes and from 25,773 to
9,999 at ten thousand.  With the header at 552 bytes — what it was when this
was measured — that is 0.52 MB saved at 1000 x 128 and 8.7 MB at 10,000 x
128, against a launch broadcast that moves ~240 MB and ~24 GB respectively.
The scatter that this entry follows saved ~98 MB at the first of those
shapes; this would save another half.

The master-side argument does not survive either.  The pre-broadcast pack
loop is real — 5.7 ms at 1000 x 128, 51 ms at 10,000 x 128, measured against
the swarm's PMIx — but it is ~22 ns per *pack call* and there are two per
process, so the cost is per process, not per buffer.  Aggregating packs the
same rank/cpuset pairs into fewer buffers and the loop takes just as long.

**So this shall not be done.**  It removes message headers, not payload, and
the headers it removes are two parts in a thousand of what the launch already
broadcasts — while adding a relay protocol on the launch path, with its own
failure handling, that has to work while the tree changes under an elastic
grow.  Fusing the slices into the xcast instead is worse, not better:
``grpcomm_xcast.c`` forwards one packed buffer shared by every child, and
says where it says so that a forward which must differ per child puts the
packing back inside the loop.

What the measurement did turn up is that the **header** was the overhead
worth attacking, and not only here: at 8 processes a node a slice message was
101 bytes of payload behind 552 bytes of header, and every RML message in the
system paid the same.  512 of those bytes were two fixed 256-byte nspace
arrays holding the same short string.  That is now fixed — the header carries
two ranks and one length-prefixed nspace, about 50 bytes on the wire — which
takes more off this traffic than aggregating it ever would have.

**Lazy proc-data registration: the withholding half.**  The *deriving*
half is in: ``dmodex_req`` answers a request for one of the placement keys
out of ``jdata->procs[rank]`` rather than going to the hosting daemon
(``prte_pmix_lazy_procdata``, on by default).  The *withholding* half is not,
and not by omission from the design — the commit that landed the derivation
does not touch ``pmix_server_register_fns.c`` at all.  That file's per-proc
loop still emits a ``PMIX_PROC_INFO_ARRAY`` for **every** proc in the job on
**every** daemon, a table that grows with the whole job on a node running a
fixed slice of it, and ``prte_hostname_cutoff`` is the record of that wall
having been met once already.  Only the two location keys are withheld, and
for a different reason (the cpuset scatter, in the section above).
Narrowing the rest is what the commit message claims and the code does not
do.

The rule it must keep to: switch on what PRRTE is the *authority* for (the
placement and binding it decided, a closed set), never on what an
application is expected to ask for.

What the derivation is worth as it stands is small, and worth knowing before
anyone spends effort on the other half.  Measured with
``contrib/dockerswarm/peerinfo.c`` — every rank asks every other rank in its
job where it is — at eight ranks over four nodes, with
``--prtemca prte_pmix_server_verbose 2``, the twenty-four peer lookups split
six answered by the derivation and eighteen by a wire round trip to the
hosting daemon; with ``prte_pmix_lazy_procdata 0`` it is zero and
twenty-four.  Every one of the six is on the master, and every one is for
``PMIX_RANK``.  That is the only key PMIx never has: it consumes the rank
entry as the array's identifier rather than storing it.  Everything else the
eager registration publishes is found locally and never reaches the
derivation, and the two keys it does not publish the derivation declines by
design (a daemon that does not fork the proc holds no cpuset, so it cannot
tell "unbound" from "not sent").  The master is the exception only because it
keeps every cpuset, so its own lookups are complete and ``PMIX_RANK`` is all
that is left to ask for.

The question this entry used to say had to be measured first — whether a
**partial** entry is usable, or whether PMIx treats a published array as the
complete answer for that rank — is answered, and the answer is that a partial
entry works.  ``pmix_server_get.c`` fetches per key; a rank present with
other keys but missing this one falls through to ``direct_modex``.  Those
eighteen wire answers above *are* that path in production: since the cpuset
scatter, a daemon publishes an entry for a remote proc with no
``PMIX_CPUSET`` in it, and the get is answered by the daemon that holds one.
The one constraint is the reverse case — for a proc the daemon **hosts**, a
missing reserved key returns ``PMIX_ERR_NOT_FOUND`` with no up-call at all,
so nothing may ever be withheld from a proc this daemon forks.

What is left to weigh is whether the withholding is worth having, because it
saves less than it appears to.  PRRTE registers ``PMIX_NODE_MAP`` and
``PMIX_PROC_MAP``, and PMIx's ``store_map`` already materializes
``PMIX_HOSTNAME``, ``PMIX_NODEID``, ``PMIX_LOCAL_RANK`` and
``PMIX_NODE_RANK`` per rank for the whole job out of them, on every daemon,
wherever the host has not spoken to that key itself.  Dropping the proc
arrays therefore does not remove the job-sized table; it removes
``PMIX_GLOBAL_RANK``, ``PMIX_APP_RANK``, ``PMIX_APPNUM``,
``PMIX_REINCARNATION``, and the three keys only the hosting daemon publishes
anyway.  It also hands the node rank to a derivation PMIx documents as
assuming this is the only job on the node, which on a shared node replaces a
right answer with a wrong one rather than with silence.

**So the withholding half shall not be done.**  It cannot remove the
job-sized per-rank table, because PMIx rebuilds most of that table from the
maps whether PRRTE publishes the proc arrays or not; what it can remove is a
handful of keys, and it pays for them with a node rank that is wrong instead
of absent wherever two jobs share a node.  The derivation already in the tree
stays — it is what lets the cpuset scatter be answered — but this entry is
closed, not deferred.  Reopen it only on a measurement showing that the table
PMIx actually builds is what hurts, and that is a PMIx question, not a PRRTE
one.
