18.3.54. pmix_server_module_t

pmix_server_module_t — The set of callback functions a host environment provides to the PMIx server library.

18.3.54.1. SYNTAX

18.3.54.1.1. C Syntax

#include <pmix_server.h>

typedef struct pmix_server_module_4_0_0_t {
    /* v1x interfaces */
    pmix_server_client_connected_fn_t   client_connected; // DEPRECATED
    pmix_server_client_finalized_fn_t   client_finalized;
    pmix_server_abort_fn_t              abort;
    pmix_server_fencenb_fn_t            fence_nb;
    pmix_server_dmodex_req_fn_t         direct_modex;
    pmix_server_publish_fn_t            publish;
    pmix_server_lookup_fn_t             lookup;
    pmix_server_unpublish_fn_t          unpublish;
    pmix_server_spawn_fn_t              spawn;
    pmix_server_connect_fn_t            connect;
    pmix_server_disconnect_fn_t         disconnect;
    pmix_server_register_events_fn_t    register_events;
    pmix_server_deregister_events_fn_t  deregister_events;
    pmix_server_listener_fn_t           listener;
    /* v2x interfaces */
    pmix_server_notify_event_fn_t       notify_event;
    pmix_server_query_fn_t              query;
    pmix_server_tool_connection_fn_t    tool_connected; // DEPRECATED
    pmix_server_log_fn_t                log;            // DEPRECATED
    pmix_server_alloc_fn_t              allocate;
    pmix_server_job_control_fn_t        job_control;
    pmix_server_monitor_fn_t            monitor;
    /* v3x interfaces */
    pmix_server_get_cred_fn_t           get_credential;
    pmix_server_validate_cred_fn_t      validate_credential;
    pmix_server_iof_fn_t                iof_pull;
    pmix_server_stdin_fn_t              push_stdin;
    /* v4x interfaces */
    pmix_server_grp_fn_t                group;
    pmix_server_fabric_fn_t             fabric;
    /* v6x interfaces */
    pmix_server_client_connected2_fn_t  client_connected2;
    pmix_server_tool_connection2_fn_t   tool_connected2;
    pmix_server_log2_fn_t               log2;
    /* pending interfaces */
    pmix_server_session_control_fn_t    session_control;
    pmix_server_resource_block_fn_t     resource_block;

} pmix_server_module_t;

18.3.54.2. DESCRIPTION

The pmix_server_module_t structure is the primary integration point between the PMIx server library and its host environment — a resource manager, launcher, or other system-management-stack (SMS) element. It is a table of function pointers, one per PMIx server operation, that the host implements and passes by pointer to PMIx_server_init(3).

PMIx uses a “function-shipping” model: each PMIx client request that requires host action is mirrored by an entry in this structure. When the PMIx server library receives such a request from a local client (or tool), it invokes the corresponding callback in the host’s module, handing the host the parameters needed to service the request. This lets a resource manager implement the server role without any knowledge of PMIx internals.

Unimplemented callbacks may be left NULL. A host is not required to support every operation. Any function the host does not provide is indicated by a NULL pointer in the structure; the library detects the NULL entry and returns PMIX_ERR_NOT_SUPPORTED to the requesting client for that operation rather than dereferencing the pointer (see, for example, pmix_server_publish in src/server/pmix_server_ops.c, which returns PMIX_ERR_NOT_SUPPORTED when NULL == pmix_host_server.publish). This behavior is documented at the top of include/pmix_server.h: “Any functions not supported by the RM can be indicated by a NULL for the function pointer. Client calls to such functions will have a ‘not supported’ error returned.”

Callbacks are asynchronous. For performance and scalability, the host is required to return from every callback as quickly as possible, performing the actual work asynchronously. Nearly all callbacks are therefore non-blocking: they accept a completion callback function (cbfunc) and an opaque cbdata pointer, and the host is expected to invoke cbfunc with the result once the operation completes. The type of the completion function varies by operation (for example, pmix_op_cbfunc_t for simple status returns, pmix_modex_cbfunc_t for fence/modex data, pmix_info_cbfunc_t for query, allocate, job-control, monitor, group, fabric, and session operations, and so on).

Return-value contract. The value a callback returns tells the library how completion will be signaled:

PMIX_SUCCESS — the request has been accepted and is being processed asynchronously. The host will invoke the supplied cbfunc later with the final result.
PMIX_OPERATION_SUCCEEDED — the operation completed immediately and successfully. The host must not invoke cbfunc; the library treats the operation as already finished.
Any other (error) value — the request failed and the host will not invoke cbfunc. The library reports the error to the client directly and releases any state it was holding for the request.

Data ownership. All data passed into a host callback is owned by the PMIx server library and must not be freed by the host. Conversely, data the host returns through a completion callback is owned by the host, which may release it once the callback returns.

18.3.54.3. ABI STABILITY

The pmix_server_module_t structure is consumed by host environments that may initialize it using positional (non-designated) initializers. Consequently the order of fields is a binary-compatibility contract and is frozen:

Existing fields are never removed and never reordered.
New callbacks are only ever appended at the end of the structure. The /* v1x interfaces */ … /* pending interfaces */ group comments record the release series in which each block of members was added.
When a callback’s signature must change, the original field is retained in place (marked // DEPRECATED) and a replacement is appended at the end with a numeric suffix on its name.

The structure currently carries three deprecation pairs. In each case the library prefers the newer member and falls back to the deprecated one only if the newer member is NULL:

client_connected → client_connected2. The original pmix_server_client_connected_fn_t takes only proc, server_object, cbfunc, and cbdata. The replacement pmix_server_client_connected2_fn_t inserts an info[]/ninfo array so additional information can be passed about the connecting client.
tool_connected → tool_connected2. The original pmix_server_tool_connection_fn_t returns void; the replacement pmix_server_tool_connection2_fn_t has an otherwise identical signature but returns a pmix_status_t so the host can synchronously reject a tool connection request.
log → log2. The original pmix_server_log_fn_t returns void; the replacement pmix_server_log2_fn_t has an identical parameter list but returns a pmix_status_t so the host can indicate synchronously whether the log request was accepted.

18.3.54.4. CALLBACK FUNCTIONS

The members are described below in structure order.

18.3.54.4.1. client_connected

client_connected (pmix_server_client_connected_fn_t) — DEPRECATED; use client_connected2. Invoked when a registered client connects to the local server. The client is held blocked until the host invokes cbfunc, allowing the library to release it. This original form conveys only the client’s proc and server_object.

18.3.54.4.2. client_finalized

client_finalized (pmix_server_client_finalized_fn_t) — Notifies the host that a local client has called PMIx_Finalize. The client is held blocked until the host invokes cbfunc. The host should release any per-client state it associated with the process via PMIx_server_register_client.

18.3.54.4.3. abort

abort (pmix_server_abort_fn_t) — Services PMIx_Abort(3). A local client has requested that a set of processes be terminated; the procs array names the targets, or a NULL array means all processes in the caller’s namespace. The requesting client is held blocked until the host invokes cbfunc. The host performs the requested termination and reports the result.

18.3.54.4.4. fence_nb

fence_nb (pmix_server_fencenb_fn_t) — Services PMIx_Fence(3) and its non-blocking form. Called once all local participants of a fence have contributed; the host collectively exchanges the supplied data blob among all servers hosting participants and returns the aggregated result through a pmix_modex_cbfunc_t. A NULL data means the local processes had nothing to contribute. Directives in the info array steer the collective and are optional unless the mandatory flag is set.

The info array may include PMIX_LOCAL_COLLECTIVE_STATUS (pmix_status_t), by which the PMIx server library reports to the host the status of the local portion of the collective — for example, an error detected among the local participants. A host that receives a failing status should propagate it into the collective result rather than proceeding with the data exchange. The library may likewise supply PMIX_LOCAL_PARTICIPANTS (pmix_data_array_t*), an array of pmix_proc_t identifying the local processes that contributed to the collective, and PMIX_SORTED_PROC_ARRAY (bool) to indicate that the array of participating processes has already been sorted into canonical order so the host need not sort it again. These attributes apply equally to the other collective module functions below (connect, disconnect, and group).

18.3.54.4.5. direct_modex

direct_modex (pmix_server_dmodex_req_fn_t) — Requests that the host contact the remote server hosting a given process and retrieve that process’s modex blob on demand (used when a PMIx_Get(3) cannot be satisfied from local data). The blob is returned through a pmix_modex_cbfunc_t. A timeout directive may be supplied to bound the wait. The request may carry a PMIX_REQUIRED_KEY (char*) naming the specific key the requester is waiting on, allowing the host to defer its response until that key has been posted by the target process.

18.3.54.4.6. publish

publish (pmix_server_publish_fn_t) — Services PMIx_Publish(3). Stores the provided key/value data in the host’s data store subject to the requested range and persistence directives, recording the publisher’s identity for later lookup. Completion is reported through a pmix_op_cbfunc_t.

18.3.54.4.7. lookup

lookup (pmix_server_lookup_fn_t) — Services PMIx_Lookup(3). Retrieves previously published data for a NULL-terminated array of string keys, honoring any wait/timeout directives, and returns the results through a pmix_lookup_cbfunc_t.

18.3.54.4.8. unpublish

unpublish (pmix_server_unpublish_fn_t) — Services PMIx_Unpublish(3). Deletes previously published data matching a NULL-terminated array of keys (within any specified range) and reports completion through a pmix_op_cbfunc_t.

18.3.54.4.9. spawn

spawn (pmix_server_spawn_fn_t) — Services PMIx_Spawn(3). Launches the given array of applications. A failure to start any process causes the entire request to be terminated and an error returned. The namespace of the spawned job is returned through a pmix_spawn_cbfunc_t. The job-level information accompanying the request may include PMIX_REQUESTOR_IS_TOOL or PMIX_REQUESTOR_IS_CLIENT (bool), indicating whether the process that issued the spawn request is a tool or a client, respectively; a host may use this to apply different policies to tool-initiated launches.

18.3.54.4.10. connect

connect (pmix_server_connect_fn_t) — Services PMIx_Connect(3). Records the specified processes as “connected” so the host treats the failure of any of them as a reportable event. This is a client-side collective, so the callback fires once all participants have contributed; completion is reported through a pmix_op_cbfunc_t. As with fence_nb, the info array may carry PMIX_LOCAL_COLLECTIVE_STATUS, PMIX_LOCAL_PARTICIPANTS, and PMIX_SORTED_PROC_ARRAY describing the local portion of the collective.

18.3.54.4.11. disconnect

disconnect (pmix_server_disconnect_fn_t) — Services PMIx_Disconnect(3). Reverses a prior connect of the same set of processes; an error is returned if the set was not previously connected. Completion is reported through a pmix_op_cbfunc_t. As with fence_nb, the info array may carry PMIX_LOCAL_COLLECTIVE_STATUS, PMIX_LOCAL_PARTICIPANTS, and PMIX_SORTED_PROC_ARRAY describing the local portion of the collective.

18.3.54.4.12. register_events

register_events (pmix_server_register_events_fn_t) — Part of the support for PMIx_Register_event_handler(3). Tells the host that the library wishes to receive notification of the specified (typically environmental) event codes; the host translates its internal codes to the corresponding PMIx codes when it later notifies. Completion is reported through a pmix_op_cbfunc_t.

18.3.54.4.13. deregister_events

deregister_events (pmix_server_deregister_events_fn_t) — The companion to register_events (see PMIx_Deregister_event_handler(3)). Cancels a prior registration for the specified event codes. The host remains obligated to report job-related events regardless.

18.3.54.4.14. listener

listener (pmix_server_listener_fn_t) — Optionally lets the host own the rendezvous listening socket. The host monitors the descriptor, accepts incoming client connections, and passes each accepted socket to the provided pmix_connection_cbfunc_t. Leaving this member NULL causes the library to spawn its own internal listener thread.

18.3.54.4.15. notify_event

notify_event (pmix_server_notify_event_fn_t) — Services PMIx_Notify_event(3) when an event generated by the server or one of its local clients must be delivered beyond the local node. The host propagates the event (identified by code, source, and range) and reports completion through a pmix_op_cbfunc_t.

18.3.54.4.16. query

query (pmix_server_query_fn_t) — Services PMIx_Query_info(3). The host answers an array of pmix_query_t requests on behalf of the identified process and returns the results through a pmix_info_cbfunc_t.

18.3.54.4.17. tool_connected

tool_connected (pmix_server_tool_connection_fn_t) — DEPRECATED; use tool_connected2. Invoked when a tool connects and needs a namespace/rank assignment. The host returns the assigned identifier through a pmix_tool_connection_cbfunc_t. This original form returns void and so cannot synchronously reject the connection.

18.3.54.4.18. log

log (pmix_server_log_fn_t) — DEPRECATED; use log2. Services PMIx_Log(3) on behalf of a client. The host must not call PMIx_Log from within this callback (doing so would create an infinite loop); instead it forwards the data to a gateway/logging channel or returns a not-supported error. Completion is reported through a pmix_op_cbfunc_t. This original form returns void.

18.3.54.4.19. allocate

allocate (pmix_server_alloc_fn_t) — Services PMIx_Allocation_request(3). The host acts on the requested allocation modification (identified by a pmix_alloc_directive_t) and returns any results through a pmix_info_cbfunc_t. The directives may include PMIX_REQUESTOR (pmix_proc_t*) identifying the process on whose behalf the request was made when it was relayed by another process.

18.3.54.4.20. job_control

job_control (pmix_server_job_control_fn_t) — Services PMIx_Job_control(3). The host executes the requested control action (e.g., pause, resume, signal, terminate) against the target processes and returns results through a pmix_info_cbfunc_t. The directives may include PMIX_REQUESTOR (pmix_proc_t*) identifying the process on whose behalf the request was made when it was relayed by another process.

18.3.54.4.21. monitor

monitor (pmix_server_monitor_fn_t) — Services PMIx_Process_monitor(3). The host arranges the requested monitoring (e.g., heartbeat or file-based) of the requestor, raising the specified error status on failure, and returns results through a pmix_info_cbfunc_t.

18.3.54.4.22. get_credential

get_credential (pmix_server_get_cred_fn_t) — Services PMIx_Get_credential(3). The host requests a security credential from the SMS on behalf of the identified process and returns it through a pmix_credential_cbfunc_t.

18.3.54.4.23. validate_credential

validate_credential (pmix_server_validate_cred_fn_t) — Services PMIx_Validate_credential(3). The host asks the SMS to validate the supplied credential and returns the result through a pmix_validation_cbfunc_t.

18.3.54.4.24. iof_pull

iof_pull (pmix_server_iof_fn_t) — Services PMIx_IOF_pull(3). Registers the local server to receive the specified IO channels (a pmix_iof_channel_t bitmask) forwarded from a set of source processes; the PMIX_IOF_STOP directive instead removes the server from the distribution list. Note that stdin cannot be pulled with this call. Completion is reported through a pmix_op_cbfunc_t.

18.3.54.4.25. push_stdin

push_stdin (pmix_server_stdin_fn_t) — Services PMIx_IOF_push(3). Passes a stdin byte object from a source process to the host for delivery to the named target processes (a WILDCARD rank targets all processes in a namespace). Completion is reported through a pmix_op_cbfunc_t.

18.3.54.4.26. group

group (pmix_server_grp_fn_t) — Services PMIx_Group_construct(3) and the related group operations. The op argument (a pmix_group_operation_t) selects construct, destruct, or cancel of the named group across its member processes; directives may request, for example, assignment of a group context ID. Results are returned through a pmix_info_cbfunc_t. As with fence_nb, the info array may carry PMIX_LOCAL_COLLECTIVE_STATUS, PMIX_LOCAL_PARTICIPANTS, and PMIX_SORTED_PROC_ARRAY describing the local portion of the collective.

18.3.54.4.27. fabric

fabric (pmix_server_fabric_fn_t) — Services PMIx_Fabric_register(3) and related fabric operations. The host retrieves fabric-related information from the server supporting the system scheduler, per the pmix_fabric_operation_t op, and returns it through a pmix_info_cbfunc_t.

18.3.54.4.28. client_connected2

client_connected2 (pmix_server_client_connected2_fn_t) — Replaces the deprecated client_connected. Invoked when a registered client connects; the library prefers this member and only falls back to client_connected when this is NULL. Adds an info[]/ninfo array carrying additional information about the connecting client. The client is held blocked until the host invokes cbfunc.

18.3.54.4.29. tool_connected2

tool_connected2 (pmix_server_tool_connection2_fn_t) — Replaces the deprecated tool_connected. Invoked when a tool connects and needs a namespace/rank assignment; the library prefers this member over tool_connected. Unlike the original, it returns a pmix_status_t so the host can synchronously reject the connection. The assigned identifier is returned through a pmix_tool_connection_cbfunc_t.

18.3.54.4.30. log2

log2 (pmix_server_log2_fn_t) — Replaces the deprecated log and services PMIx_Log(3); the library prefers this member and falls back to log only when it is NULL. Identical in parameters to log but returns a pmix_status_t. As with log, the host must not call PMIx_Log from within this callback.

18.3.54.4.31. session_control

session_control (pmix_server_session_control_fn_t) — Services PMIx_Session_control(3). The host executes the requested control operation against the identified session (sessionID) and returns results through a pmix_info_cbfunc_t. The directives may include PMIX_REQUESTOR (pmix_proc_t*) identifying the process on whose behalf the request was made when it was relayed by another process.

18.3.54.4.32. resource_block

resource_block (pmix_server_resource_block_fn_t) — Services PMIx_Resource_block(3). The host defines, deletes, extends, or removes resources from a named resource block per the pmix_resource_block_directive_t and the supplied array of pmix_resource_unit_t units. Completion is reported through a pmix_op_cbfunc_t.