8.8.1. Reaching a data server in another DVM

8.8.1.1. What this is for

PRRTE’s data server backs PMIx’s publish/lookup service. Normally the store lives on this DVM’s own master and every participant reaches it over the RML — see src/runtime/data_server/AGENTS.md.

prte_pmix_server_uri asks for something else: that the store live in a different DVM, so that jobs launched by different invocations can find each other’s data. That is what an MPI application is doing when one mpirun calls MPI_Publish_name and another calls MPI_Comm_accept, and it is why pmix_server_register_fns.c publishes a job’s info whenever an external server is configured: any subsequent connect has to be able to retrieve it.

8.8.1.2. Why it could not work over the RML

The RML addresses a peer by rank, and stamps the sender’s own namespace on it. Every send entry point takes a pmix_rank_t (prte_rml_send_buffer_nb and its variants), send_buffer() builds the destination as PMIX_LOAD_PROCID(&snd->dst, PRTE_PROC_MY_NAME->nspace, rank), and prte_oob_base_send_nb resolves the next hop the same way. A daemon of another DVM is therefore not nameable: the namespace parsed out of the server’s URI was discarded and the request went to this DVM’s rank 0.

That has been so since the March 2022 RML rework (17368b1b60), which rewrote the one line that mattered:

-    PRTE_RML_SEND(rc, target, xfer, PRTE_RML_TAG_DATA_SERVER);
+    PRTE_RML_SEND(rc, target->rank, xfer, PRTE_RML_TAG_DATA_SERVER);

It is not something the transport can be talked into. Below the API, everything is indexed by rank within this DVM — prte_rml_is_node_up(), prte_rml_get_route(), the boot-epoch table — so a foreign rank collides with a local one at every layer, not just in the wire header. Two further gaps made the feature unreachable even in principle: the reply was addressed with sender->rank in the server’s namespace, and nothing published a URI in the form the OOB parses (--report-uri emits the PMIx server URI, nspace.rank;tcp4://host:port, while process_uri wants PRRTE’s own tcp://ip:port:ifmask).

8.8.1.3. The approach: a PMIx tool connection

Rather than teach the RML to carry a second namespace, the crossing is made where PMIx already supports one. The DVM master attaches to the remote DVM’s PMIx server as a tool and reissues the operation as an ordinary PMIx_Publish/Lookup/Unpublish. The remote DVM’s own upcalls then reach its data server exactly as a local client’s would.

This is not a new mechanism in PRRTE: prte_pmix_set_scheduler() already attaches the master to a scheduler the same way.

Consequences, all of them wanted:

  • The RML and the wire header stay as they are. Nothing has to name a foreign process, so the OOB header can go on carrying one namespace — see the entry in Deferred work.

  • The URI problem disappears. What PMIx_tool_attach_to_server wants under PMIX_SERVER_URI is exactly what prte --report-uri writes.

  • One connection per DVM. Only the master attaches; every other daemon relays to it over the RML, as it already does for a locally-hosted data server.

app proc ──PMIx──▶ its prted ──RML(DATA_SERVER)──▶ this DVM's master
                                                          │
                                                 prte_ds_relay()
                                                          │
                                         PMIx tool connection
                                                          ▼
                                          the server DVM's master
                                                          │
                                        its own publish/lookup upcall
                                                          ▼
                                               its data server

8.8.1.4. Two things the design has to get right

Whose request is it. A relayed publish arrives at the far end as an operation by this daemon’s tool identity, so every item would be owned by the relay rather than by the process that asked. Ownership is not cosmetic: it decides who may unpublish an item, what PMIX_RANGE_NAMESPACE admits, and which items a purge takes when a job ends. The requesting process is therefore carried in PMIX_REQUESTOR, which the PMIx Standard defines for precisely this — “used when relaying a request to the PMIx library on behalf of someone else where the API doesn’t include a requestor parameter”.

The far end honors it only from a tool (prte_ds_check_requestor()). Relaying is what a tool does; an application process claiming to act for another has no such standing, and allowing it would let any process publish — and unpublish — under a peer’s identity.

Which server is primary. PMIx directs a tool’s client-side calls at whichever attached server is currently primary, and only one may be primary at a time. A master can hold two connections — a scheduler and a data server — so an operation may not assume the primary is whatever the last operation left in place. prte_pmix_set_primary_server() is the single point that designates one, and both the data-server relay and prte_pmix_set_scheduler() call it before every operation. It is a no-op when the named server is already primary, and it is race-free because the PMIx client calls pack and send synchronously, on the same thread, before returning.

That replaces the sticky scheduler_set_as_server flag, which recorded “the scheduler has been made primary” once and would have been wrong the moment a second connection existed.

8.8.1.5. What changed

file

change

src/prted/pmix/pmix_server_pub.c

init_server() attaches as a tool instead of parsing an RML URI; only the master attaches; execute() sends to the master when a server is external; the purge command packs its directives

src/runtime/data_server/ds_relay.c

new — the relay itself

src/runtime/data_server/ds_main.c

dispatch to the relay; prte_ds_check_requestor()

src/runtime/data_server/ds_*.c

honor PMIX_REQUESTOR in the directive scan

src/prted/pmix/pmix_server_allocate.c

prte_pmix_set_primary_server()

src/mca/state/{dvm,base}

purge senders match the new packing, and address the master when the server is external

The purge command’s wire format gained a directive array. That is allowed without ceremony — the message is exchanged inside one DVM, whose processes all come from one build — but both ends change in the same commit: three senders (pmix_server_unpublish_fn, state_dvm.c, state_base_fns.c) and one reader (ds_purge.c).

8.8.1.6. How it is tested

Nothing about this exists on a single DVM, so the coverage is in contrib/dockerswarm’s test_runtime phase: three DVMs at once — a server and two clients pointed at it — asserting that a key published in one client DVM is found in the other, that the answer names the publishing process and not the relay, that a PMIX_WAIT lookup parked from one DVM is woken by a publish from the other, that an ended job’s data is purged from the external server, that the purge takes only that job’s data, and — the control — that a DVM which was not given the URI sees none of it.