.. _rml-label:

The Runtime Messaging Layer (RML)
=================================

The Runtime Messaging Layer is how PRRTE daemons talk to one another.  Every
control message that crosses the DVM — launch commands, I/O forwarding,
collective operations, PMIx data exchanges, fault notices — travels over the
RML.  It provides a single service: *deliver this buffer to that daemon, under
this tag*, asynchronously and in order per connection.

This page explains how the RML is put together and how a message flows through
it.  The code lives entirely in ``src/rml/``; a shorter, editing-oriented map
is in ``src/rml/AGENTS.md``.  Two subsystems are large enough to have their own
pages:

.. toctree::
   :maxdepth: 1

   oob
   relm

This page covers the RML core (the send/receive API, message matching, and the
radix routing tree).  :ref:`rml-oob-label` covers the TCP transport in detail,
and :ref:`rml-relm-label` covers reliable messaging.

Historical note: one directory, once three frameworks
-----------------------------------------------------

``src/rml`` was originally **three** MCA frameworks:

* **rml** — the messaging API,
* **routed** — pluggable routing-tree algorithms, and
* **oob** ("out of band") — pluggable transports, each with swappable modules.

PRRTE only ever had one RML, one routing algorithm (a radix tree), and one
transport (TCP), so the three frameworks were collapsed into the single
``src/rml`` directory.  The collapse merged the files but, for a long time,
left the multi-component *abstractions* in place — dead failover paths, a relay
macro nobody called, a routed-module name in every wire header, and comments
describing a world of many components, modules, and transports.  That
scaffolding has since been removed.  When you read the code today there is one
path: **RML API → radix routing → TCP transport**, plus a newer fault-tolerance
layer layered on top.

Architecture and key state
--------------------------

Two global structures hold essentially all RML state, and both are owned by the
PRRTE progress thread:

``prte_rml_base`` (``src/rml/rml.h``, defined in ``rml.c``)
    The messaging and routing state.  It holds the two matching lists —
    ``posted_recvs`` (receives the local code has posted) and
    ``unmatched_msgs`` (messages that arrived before a matching receive was
    posted) — plus the routing-tree state: the number of daemons ``n_dmns``,
    the ``failed_dmns``/``global_failed_dmns`` bitmaps, this daemon's
    ``ancestors`` and ``children`` in the tree, and its ``lifeline`` (immediate
    parent).

``prte_oob_base`` (``src/rml/oob/oob.h``, defined in ``oob_tcp.c``)
    The transport state: the list of known ``peers`` (each with its addresses,
    socket, and send queue), the local network interfaces, listener sockets,
    and the many TCP tuning MCA parameters (buffer sizes, keepalives, retry
    limits, ``max_msg_size``).

The message objects (``src/rml/rml_types.h``) are reference-counted PMIx
objects:

* ``prte_rml_send_t`` — an outbound message (destination, origin, tag, data
  buffer, retry count).
* ``prte_rml_recv_t`` — an inbound message waiting to be matched/delivered.
* ``prte_rml_posted_recv_t`` — a standing receive: a (peer, tag, persistent,
  callback) tuple.

Everything runs on one thread.  Work that originates elsewhere is moved onto
the progress thread with a *caddy* — a small heap object carrying the request
plus a ``pmix_event_t`` — posted via ``PRTE_PMIX_THREADSHIFT``.  ``PRTE_OOB_SEND``
and ``prte_rml_recv_buffer_nb`` are the two entry points that do this.

Sending a message
-----------------

The public entry point is the ``PRTE_RML_SEND(rc, dst, buffer, tag)`` macro,
which calls ``prte_rml_send_buffer_nb`` (``src/rml/rml_send.c``).  The RML takes
ownership of ``buffer``.

#. **Validation and self-sends.**  Invalid tag/rank or a known-down destination
   are rejected immediately.  A message addressed to *this* daemon never touches
   the network: it is wrapped in a ``prte_rml_recv_t`` and re-posted locally with
   ``PRTE_RML_ACTIVATE_MESSAGE``, so it enters the same delivery path as a
   received message.

#. **Hand to the transport.**  Otherwise the call builds a ``prte_rml_send_t``
   (``dst``, ``origin`` = me, ``tag``, ``dbuf``) and invokes ``PRTE_OOB_SEND``,
   which thread-shifts to ``prte_oob_base_send_nb`` (``oob_base_stubs.c``).

#. **Route and connect.**  ``prte_oob_base_send_nb`` first drops the message if
   the destination is down or the retry budget (``prte_rml_max_retries``) is
   exhausted, reporting the failure back through the send callback.  Otherwise it
   computes the **next hop** with ``prte_rml_get_route(dst)`` and looks up the
   TCP peer for that hop.  If the peer is unknown, it obtains the peer's contact
   URI (directly for the HNP, or from the PMIx store) and builds a peer object.

#. **Queue.**  If the peer is already connected, the message is queued for
   immediate transmission (``MCA_OOB_TCP_QUEUE_SEND``); otherwise it is queued as
   pending (``MCA_OOB_TCP_QUEUE_PENDING``) and the connection state machine in
   ``oob_tcp_connection.c`` is started.  Once the socket is up and the IDENT
   handshake has completed, the send handler in ``oob_tcp_sendrecv.c`` writes the
   header and then the payload.

Receiving, matching, and relaying
---------------------------------

On the wire, every message carries a ``prte_oob_tcp_hdr_t``
(``oob/oob_tcp_hdr.h``): the ``origin``, the final ``dst``, the ``tag``, a
sequence number, the payload length, and a message ``type`` (``IDENT``/``PROBE``
for the handshake, ``USER`` for a normal message).  The header is exchanged only
between daemons of the same DVM — all running the same build — so its layout is
not a stable ABI.

When a peer's socket has delivered a complete message, the recv handler
(``oob_tcp_sendrecv.c``) inspects ``hdr.dst``:

* **For us.**  It calls ``PRTE_RML_POST_MESSAGE``, which thread-shifts to
  ``prte_rml_base_process_msg`` (``rml_base_msg_handlers.c``).  That handler walks
  ``posted_recvs`` looking for a receive whose (peer, tag) matches — using
  ``PMIX_CHECK_PROCID`` so wildcard receives work — and fires its callback.  A
  non-persistent receive is removed once it fires.  If nothing matches, the
  message is parked on ``unmatched_msgs``.

* **Not for us.**  This daemon is an intermediate hop.  The handler rebuilds a
  ``prte_rml_send_t`` with the same ``origin``/``dst`` and re-enters
  ``PRTE_OOB_SEND``, which routes the message on toward the next hop.  Relaying is
  therefore just "send again from here."

The dual of ``process_msg`` is ``prte_rml_base_post_recv``: when new code posts a
receive, that handler also scans ``unmatched_msgs`` so that any message which
arrived early is delivered right away.  ``prte_rml_purge`` clears both lists for a
peer that has gone away.

Routing: the radix tree
-----------------------

Daemons are numbered ``0 .. N-1`` (rank 0 is the HNP) and arranged in a radix
tree.  The radix (fan-out) defaults to 64 and is set with
``--prtemca rml_base_radix`` (``--prtemca routed_radix`` is a deprecated
synonym).  The minimum is 2 — the tree arithmetic divides by ``radix - 1``, so
a smaller value is not a degenerate tree but a division by zero, and a value
below 2 is clamped with a warning.  The tree keeps the number of hops between
any two daemons small while bounding each daemon's connection count.

``prte_rml_get_route(target)`` (``routed_radix.c``) returns the rank of the next
hop:

* ``target`` itself if it is this daemon;
* this daemon's parent (its ``lifeline``) if ``target`` lies outside this
  daemon's subtree — i.e., the message must go *up* the tree;
* otherwise the child whose subtree contains ``target`` — the message goes
  *down* toward it.

All of the arithmetic — parent, child, sibling, "does this subtree contain rank
r," "which child subtree holds r," moving up and down by depth, and "next living
rank" traversals that skip failed daemons — lives as pure functions over rank
numbers in ``radix.h``.  That header is deliberately self-contained and includes
``radix_node_is_valid`` as an executable statement of the tree invariants; it is
the first thing to read if you need to modify the routing math.

Fault tolerance and reliable messaging
--------------------------------------

The routing tree is not static: daemons can fail, and (in elastic mode) the DVM
can grow and shrink.  ``prte_rml_base`` therefore tracks which daemons have
failed and repairs the tree in place.

When a connection is lost (``prte_rml_route_lost``) or a fault notice arrives,
``prte_rml_repair_routing_tree`` (``routed_radix.c``) recomputes this daemon's
ancestors and children, works out whether this daemon has been *promoted* to
fill a hole higher up, and packages the before/after difference into a
``prte_rml_recovery_status_t``.  Faults are handled twice — once **locally** as
they are discovered and once **globally** when the HNP broadcasts the confirmed
set — so handlers can distinguish "recover now" from "everyone agrees this is
final."  The recovery status is then delivered to the registered fault handlers:
the RML's own (``rml_fault_handler.c``, which drives process states and
death/adoption notices), followed by grpcomm, filem, and relm.

Both of those notices carry the sender's view of the *other* daemon's ancestor
list, and the receiver reconciles it against its own before acting.  That is
what lets two daemons agree on the repaired tree in one exchange rather than
waiting for a message addressed to a dead rank to time out: the failure notice
a re-homed daemon sends upward reports only the failures inside its own
subtree, and the ancestor whose death moved it is never in that set, so without
the lineage a new parent has no way to learn why it suddenly has a child.  A
report that cannot be reconciled is fatal coming *down* the tree — an adoption
notice is how a daemon learns its own place — and merely dropped coming up,
where the same news always arrives by another route.

``relm`` (the ``src/rml/relm/`` subtree) is the **reliable messaging** layer.
``prte_rml_send_buffer_reliable_nb`` routes through it; it drives a small state
machine that re-sends messages over the repaired tree so that a message in
flight when a daemon dies is not simply lost.  It is newer than the collapsed
core and is intentionally kept as its own module.  Its protocol — message
states, acknowledgements, replay, and the link-update recovery dance — is
described on its own page: :ref:`rml-relm-label`.

Elastic growth and launcher-less bootstrap
------------------------------------------

The tree is not fixed for the life of the DVM.  In *elastic* mode it can grow
and shrink, and in a *bootstrapped* DVM the daemons come up independently rather
than being fanned out by a launcher.  Both add behavior to the routing and
transport layers:

* **A permanent departure set.**  ``prte_rml_base`` carries a third failure
  bitmap, ``dead_dmns``, alongside ``failed_dmns`` and ``global_failed_dmns``.
  The first two are re-initialized on every ``prte_rml_compute_routing_tree``;
  ``dead_dmns`` is constructed once and never reset, so a daemon that shrank out
  (or failed) stays a permanent hole across the recompute a subsequent grow
  triggers.  ``compute_routing_tree`` restores the dead marks into the
  freshly-wiped ``failed_dmns`` before rebuilding the tree, so the rebuilt tree
  routes around the hole instead of addressing a reused-looking but dead vpid.
  (The DVM never reuses a daemon vpid.)

* **The departure set reaches a new daemon at launch, not by message.**  A
  daemon started into a DVM that has already shrunk needs those holes *before*
  it computes its first routing tree, and no message can deliver them: with the
  wrong tree, everything it sends upward — including the warm-up that asks for
  the nidmap that would correct it — is addressed to a retired vpid nothing can
  contact, so it never learns, never reports in, and every job placed on it
  hangs.  The launcher therefore renders the set onto the ``prted`` command
  line as ``rml_base_dead_dmns`` — a range-collapsed list such as ``2,7:9``,
  colon-separated because a released PMIx refuses an MCA value whose second
  character is a dash — and the daemon adopts it in ``prte_rml_open`` before
  the first tree is built.  The daemon this strands is the one whose
  fault-free radix parent *is* a retired vpid, so at the default radix, where
  every daemon's parent is the HNP, the problem does not arise; it needs a
  tree with depth.

* **A leaving daemon departs on the first lost route.**  When
  ``prte_dvm_leaving`` is set — this daemon was named as a shrink target —
  ``prte_rml_route_lost`` terminates immediately on *any* dropped connection
  rather than trying to recover, so a departing daemon's normal disconnects are
  never mistaken for faults.

* **Bootstrap synthesizes contact information on demand.**  With no nidmap to
  distribute URIs, the OOB derives a peer's contact URI from configuration when
  it first needs to reach it, tolerates a synthesized URI that lacks an
  interface mask, and treats a parent that is not yet up as a heal-to-ancestor
  event rather than a fatal failure.  The details live in :ref:`rml-oob-label`.

Where to look
-------------

* Sending: ``rml_send.c`` → ``oob/oob_base_stubs.c`` → ``oob/oob_tcp_connection.c``
  → ``oob/oob_tcp_sendrecv.c``.
* Receiving/matching: ``oob/oob_tcp_sendrecv.c`` → ``rml_base_msg_handlers.c``.
* Routing math: ``routed_radix.c`` and ``radix.h``.
* Fault handling: ``routed_radix.c`` (``repair_routing_tree``),
  ``rml_fault_handler.c``, and ``relm/``.
* Editing map and gotchas: ``src/rml/AGENTS.md``.
