20.2.249. PMIx_server_register_nspace

PMIx_server_register_nspace — Register a namespace and its job-level information with the PMIx server library.

20.2.249.1. SYNOPSIS

#include <pmix_server.h>

pmix_status_t PMIx_server_register_nspace(const pmix_nspace_t nspace, int nlocalprocs,
                                          pmix_info_t info[], size_t ninfo,
                                          pmix_op_cbfunc_t cbfunc, void *cbdata);

20.2.249.1.1. Python Syntax

from pmix import *

foo = PMIxServer()
# ... after a successful foo.init(pydirs, map) ...
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_UNIV_SIZE,
           'value': 4, 'val_type': PMIX_UINT32},
          {'key': PMIX_JOB_SIZE,
           'value': 4, 'val_type': PMIX_UINT32}]
rc = foo.register_nspace("myjob", 4, pydirs)

20.2.249.2. INPUT PARAMETERS

nspace: The namespace (a character array of maximum length PMIX_MAX_NSLEN) of the job being registered.
nlocalprocs: The number of processes from this namespace that will be launched locally — i.e., that will connect to this PMIx server. This count is required so that the server library can correctly determine when a collective operation is locally complete, even if the collective is called before all local processes have started.

A negative value has a distinct meaning: it declares the call to be an update to a namespace the library already holds, rather than the registration of a new one. See Updating a registered namespace.
info: Pointer to an array of pmix_info_t(5) structures conveying the session-, job-, application-, node-, and process-realm information for the namespace (see DIRECTIVES). A NULL value is supported when no data is to be registered (for example, when using the PMIX_REGISTER_NODATA directive).
ninfo: Number of elements in the info array.
cbfunc: Callback function of type pmix_op_cbfunc_t invoked when the registration completes. A NULL value makes the call blocking (see DESCRIPTION).
cbdata: Opaque pointer that is passed, unmodified, to cbfunc.

20.2.249.3. DESCRIPTION

Register a namespace (job) with the PMIx server library so that its job-level information can be provided to the processes in that job as they connect. The PMIx connection procedure gives the host PMIx server an opportunity to pass job-related information down to each child process — for example, the number of processes in the job, the relative local ranks of the processes, and the node and process maps. The host is free to determine which of the supported elements it provides; the defined values are described in DIRECTIVES.

The host must register every namespace that will participate in collective operations involving local processes — even a namespace from which this server hosts no local processes — if any local process might at some point perform a collective operation involving one or more processes from that namespace. This is required so the collective can determine when it is locally complete.

20.2.249.3.1. Blocking and non-blocking forms

PMIx_server_register_nspace supports both a non-blocking and a blocking mode of operation, selected by the cbfunc argument. This function is the canonical example of the PMIx thread-shifting pattern: in both modes the request parameters are packaged and posted to the library’s internal progress thread, where the actual registration work is performed.

When cbfunc is non-NULL, the call is non-blocking: the function posts the request to the progress thread and returns PMIX_SUCCESS immediately, and the provided cbfunc is invoked with the final status once registration completes. As with all non-blocking PMIx APIs, the caller must keep the info array valid until cbfunc has been invoked.

When cbfunc is NULL, the call is blocking: the library substitutes an internal callback, posts the request, and waits for the progress thread to finish before returning. In this case the result is carried by the return value itself, and on success the function returns PMIX_OPERATION_SUCCEEDED to indicate that the operation completed inline and no callback will fire.

20.2.249.3.2. Updating a registered namespace

A job’s description is not always fixed for the life of the job. Resources may be added to it, a value the host computed early may be refined, a node may be lost. This API is the supported way to revise the job-level data of a namespace that is already registered, and it is the method host environments should use.

Pass a negative nlocalprocs. The info array is then read as a revision of what the library holds for nspace rather than as a fresh registration, and no local process count is implied or changed.

The array may take either of two shapes, and they are treated identically:

only the values that changed, or
the whole description, with some values in it different from before.

Restating a value that has not changed is explicitly permitted and costs nothing: each datastore compares every entry against what it already answers for that key and discards the ones that match, so a host that periodically restates its full job description does not accumulate anything. Only entries that are new, or whose value differs, are stored and pushed to clients. This is not merely an optimization — the shared-memory datastore cannot reclaim what it has published, so a restatement that was treated as a change would grow a job’s footprint for the life of the job.

Values are applied at the job level (they are stored against PMIX_RANK_WILDCARD) and are pushed to every local client of that namespace that is already running, so a process that called PMIx_Init(3) before the update will see the revised value from its own datastore without having to ask the server for it.

A PMIX_PROC_INFO_ARRAY in an update revises the data of the single process that array describes, and a PMIX_JOB_INFO_ARRAY is unwrapped and its contents applied as job-level values.

Note

There is currently no way to remove a key through an update. An update may add a key or change the value of an existing one; a key once registered remains registered for the life of the namespace. Use PMIx_server_deregister_resources(3) for values that were registered as non-namespace resources.

20.2.249.4. DIRECTIVES

Information is passed through the info array, organized by data realm. Realm information may be passed either as individual pmix_info_t(5) entries or grouped inside a pmix_data_array_t labeled with the corresponding *_INFO_ARRAY attribute. The following attributes are required to be supported by all PMIx libraries for use with this API:

PMIX_REGISTER_NODATA (bool) — the registration is for the namespace only; do not copy any job data. Used to pre-register a namespace whose data will follow later.
PMIX_SESSION_INFO_ARRAY (pmix_data_array_t*) — an array of session-realm information for the session containing this job.
PMIX_JOB_INFO_ARRAY (pmix_data_array_t*) — an array of job-realm information for this namespace.
PMIX_APP_INFO_ARRAY (pmix_data_array_t*) — an array of application-realm information; required (one array per application) when the job contains more than one application.
PMIX_PROC_INFO_ARRAY (pmix_data_array_t*) — an array of process-realm information for a process in the job.
PMIX_NODE_INFO_ARRAY (pmix_data_array_t*) — an array of node-realm information for a node participating in the job.

20.2.249.4.1. Session-realm information

PMIX_UNIV_SIZE (uint32_t) — number of process slots allocated to the session.
PMIX_MAX_PROCS (uint32_t) — maximum number of processes; must be provided if PMIX_UNIV_SIZE is not given.
PMIX_SESSION_ID (uint32_t) — session identifier; required whenever the PMIx server library may host multiple sessions.

20.2.249.4.2. Job-realm information

PMIX_NSPACE (char*) — namespace of the job being registered.
PMIX_JOBID (char*) — job identifier assigned by the resource manager.
PMIX_JOB_SIZE (uint32_t) — total number of processes in the job.
PMIX_NODE_MAP (char*) — regular-expression representation of the nodes hosting the job (see PMIx_generate_regex2).
PMIX_PROC_MAP (char*) — regular-expression representation of the process-to-node mapping.
PMIX_NODE_MAP_RAW (char*) — comma-delimited list of the nodes containing processes for this job, provided as an alternative to the regular-expression form in PMIX_NODE_MAP.
PMIX_PROC_MAP_RAW (char*) — semicolon-delimited list of strings, each a comma-delimited list of the ranks on the corresponding node, provided as an alternative to the regular-expression form in PMIX_PROC_MAP.
PMIX_ANL_MAP (char*) — process mapping in ANL notation, as used by PMI-1 and PMI-2.
PMIX_APP_MAP_TYPE (char*) — type of mapping used to lay out the application (e.g., cyclic).
PMIX_APP_MAP_REGEX (char*) — regular expression describing the result of the mapping.
PMIX_JOB_NUM_APPS (uint32_t) — number of applications in the job; required when the job contains more than one application.
PMIX_SERVER_NSPACE (char*) — namespace of the PMIx server itself.
PMIX_SERVER_RANK (pmix_rank_t) — rank of the PMIx server itself.

The host environment is expected to supply a broad range of additional session-, job-, application-, node-, and process-realm information as required by the job. See the “Reserved Keys” chapter of the PMIx Standard for the full list and for the rules governing how each key is retrieved.

20.2.249.5. CALLBACK FUNCTION

When cbfunc is provided, it has the signature pmix_op_cbfunc_t:

typedef void (*pmix_op_cbfunc_t)(pmix_status_t status, void *cbdata);

The library invokes cbfunc from its progress thread once registration completes. status is PMIX_SUCCESS if the namespace and its data were registered successfully, or a negative PMIx error constant otherwise. cbdata is the opaque pointer passed to PMIx_server_register_nspace, allowing the host to correlate the completion with its originating request.

20.2.249.6. RETURN VALUE

For the non-blocking form (cbfunc is non-NULL), a return of PMIX_SUCCESS indicates only that the request was accepted for processing; the final status is delivered to cbfunc.

For the blocking form (cbfunc is NULL), the return value carries the result directly:

PMIX_OPERATION_SUCCEEDED — the registration completed successfully inline; no callback is or will be invoked.
PMIX_ERR_INIT — the PMIx server library has not been initialized.
PMIX_ERR_NOT_AVAILABLE — the operation cannot be serviced because the library’s progress engine has been stopped.
PMIX_ERR_BAD_PARAM — nspace is NULL or empty.
PMIX_ERR_WOULD_BLOCK — the blocking form was called from the library’s own progress thread (for example, from inside a pmix_server_module_t(5) upcall), where waiting for the request would be waiting for the caller. Use the non-blocking form from that context.

Both forms return PMIX_ERR_INIT, PMIX_ERR_NOT_AVAILABLE and PMIX_ERR_BAD_PARAM directly, since they are detected before the request is accepted.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

20.2.249.7. NOTES

There is no requirement that a namespace be registered before its clients are registered; if PMIx_server_register_client(3) is called for an as-yet-unregistered namespace, the library creates a placeholder namespace in anticipation of the eventual PMIx_server_register_nspace call. However, collective operations cannot complete locally until the namespace has been registered with its nlocalprocs count.

Large PMIX_NODE_MAP and PMIX_PROC_MAP values are commonly generated in compressed form using PMIx_generate_regex2 prior to registration.

20.2.249.8. PROGRESS THREAD RESTRICTION

A blocking PMIx call must not be made from within the PMIx progress thread. Any code the library itself invokes runs on that thread: an event handler registered through PMIx_Register_event_handler(3), a callback passed to a non-blocking PMIx API, and — in a server or tool — the completion of a host-module up-call. A blocking call waits for work that the progress thread has to perform, so making one from that thread waits for itself and never returns. The PMIx Standard disallows it, and there is no way for an implementation to service such a request.

Where this call has a blocking form — including the blocking behavior a non-blocking entry point adopts when it is passed a NULL cbfunc — that form detects the situation and returns PMIX_ERR_WOULD_BLOCK immediately, accompanied by a diagnostic naming the call. Nothing is done and no callback is invoked.

PMIX_ERR_WOULD_BLOCK here is not a transient condition to retry: it reports a call that cannot be serviced from where it was made. Reissue it as the non-blocking form with a callback, or from a thread of your own.