18.2.12. PMIx_Spawn

PMIx_Spawn, PMIx_Spawn_nb — Spawn a new job consisting of one or more applications.

18.2.12.1. SYNOPSIS

#include <pmix.h>

pmix_status_t PMIx_Spawn(const pmix_info_t job_info[], size_t ninfo,
                         const pmix_app_t apps[], size_t napps,
                         pmix_nspace_t nspace);

pmix_status_t PMIx_Spawn_nb(const pmix_info_t job_info[], size_t ninfo,
                            const pmix_app_t apps[], size_t napps,
                            pmix_spawn_cbfunc_t cbfunc, void *cbdata);

18.2.12.1.1. Python Syntax

from pmix import *

foo = PMIxClient()
# ... after a successful foo.init() ...
# jobInfo is a list of Python ``pmix_info_t`` dictionaries
jobInfo = [{'key': PMIX_NOTIFY_COMPLETION,
            'value': {'value': True, 'val_type': PMIX_BOOL}}]
# pyapps is a list of Python ``pmix_app_t`` dictionaries; each app
# provides at least a command and its argv, plus optional per-app info
pyapps = [{'cmd': "hello", 'argv': ["hello", "world"],
           'maxprocs': 4, 'info': []}]
rc, nspace = foo.spawn(jobInfo, pyapps)

18.2.12.2. INPUT PARAMETERS

job_info: Pointer to an array of pmix_info_t(5) structures conveying job-level directives that apply to the spawned job as a whole (see DIRECTIVES). A NULL value is supported when no job-level directives are desired.
ninfo: Number of elements in the job_info array.
apps: Pointer to an array of pmix_app_t structures, one per application to be launched. Each pmix_app_t describes a single application via the following fields:
- cmd (char*) — the executable to launch.
- argv (char**) — the NULL-terminated argument vector.
- env (char**) — a NULL-terminated array of environment variables (in name=value form) to be set for this application.
- cwd (char*) — the working directory in which to start the application’s processes.
- maxprocs (int) — the number of processes to start for this application.
- info / ninfo — an array of pmix_info_t(5) structures conveying per-application directives (see DIRECTIVES), and the number of elements in it.
At least one of cmd or argv must be provided in every element; an element that supplies neither causes the operation to fail with PMIX_ERR_BAD_PARAM.
napps: Number of elements in the apps array.

The non-blocking form replaces the nspace output parameter with a callback:

cbfunc: Callback function of type pmix_spawn_cbfunc_t invoked when the applications have been launched (or the launch has failed).
cbdata: Opaque pointer that is passed, unmodified, to cbfunc.

18.2.12.3. OUTPUT PARAMETERS

nspace (blocking form): A pmix_nspace_t (a fixed-size character array of at least PMIX_MAX_NSLEN + 1 bytes) into which the namespace assigned to the newly spawned job is copied on success. Because pmix_nspace_t is an array, this parameter cannot itself be NULL.

For the non-blocking form, the assigned namespace is instead delivered as the nspace argument to cbfunc. That value is not protected after the callback returns, so the receiver must copy it if it needs to be retained.

18.2.12.4. DESCRIPTION

Spawn a new job consisting of the applications described in the apps array. PMIx_Spawn is the blocking form: it does not return until the job has been launched (or the operation fails), at which point the namespace assigned to the new job is returned in nspace. PMIx_Spawn_nb is the non-blocking form: it returns immediately, and the provided cbfunc is invoked with the final status and the assigned namespace once the launch completes.

Each element of the apps array describes one application (multiple elements therefore describe an MPMD job launched as a single unit). Directives that apply to the job as a whole are passed in job_info; directives that apply to only a single application are passed in that application’s info array. Where an attribute is meaningful at both levels, a value in an application’s info array applies to that application only, while a value in job_info applies to every application in the job.

By default, the spawned processes will be PMIx “connected” to the parent process upon successful launch (see PMIx_Connect(3) for details). In practice this means the parent process is given a copy of the new job’s job-level information — so it can query that information without incurring communication penalties — and that both the parent and the members of the child job receive notification of errors arising anywhere in their combined assemblage.

Behavior of individual resource managers may differ, but it is expected that failure of any application process to start will result in termination and cleanup of all processes in the newly spawned job and return of an error code to the caller.

As with all non-blocking PMIx APIs, callers of PMIx_Spawn_nb must keep the job_info and apps arrays (and everything they reference) valid until cbfunc is invoked.

18.2.12.5. DIRECTIVES

PMIx libraries are not required to directly support any attributes for this operation; any provided attributes are passed to the host environment for processing. Unless otherwise noted, each attribute below may be placed either in the job_info array (to apply to the whole job) or in the info array of an individual pmix_app_t (to apply to that application only).

Host environments are required to support the following attributes when present:

PMIX_WDIR (char*) — working directory in which to start the spawned processes.
PMIX_SET_SESSION_CWD (bool) — set the application’s current working directory to the session working directory assigned by the RM.
PMIX_PREFIX (char*) — prefix directory to use when looking for the application’s executables and libraries.
PMIX_HOST (char*) — comma-delimited list of hosts to use for the spawned processes.
PMIX_HOSTFILE (char*) — hostfile naming the resources to use for the spawned processes.

The following attributes are optional for host environments that support this operation. Placement, mapping, and resource selection:

PMIX_ADD_HOST (char*) / PMIX_ADD_HOSTFILE (char*) — add the named hosts, or the hosts named in the given hostfile, to the existing allocation.
PMIX_PRELOAD_BIN (bool) — preload the application binaries onto the target nodes.
PMIX_PRELOAD_FILES (char*) — comma-delimited list of files to pre-position on the target nodes.
PMIX_PERSONALITY (char*) — name of the programming-model personality to assume for the application (e.g., "ompi").
PMIX_MAPBY (char*) / PMIX_RANKBY (char*) / PMIX_BINDTO (char*) — mapping, ranking, and binding policies for the spawned processes.
PMIX_PPR (char*) — number of processes to spawn on each identified resource.
PMIX_CPUS_PER_PROC (uint32_t) — number of CPUs to assign to each rank.
PMIX_CPU_LIST (char*) — list of CPUs to use for this job.
PMIX_NO_PROCS_ON_HEAD (bool) — do not place any processes on the head node.
PMIX_NO_OVERSUBSCRIBE (bool) — do not oversubscribe the CPUs.
PMIX_COLOCATE_PROCS (pmix_data_array_t*) — array of pmix_proc_t identifying the processes with which the new job’s processes are to be colocated.
PMIX_COLOCATE_NPERPROC (uint16_t) / PMIX_COLOCATE_NPERNODE (uint16_t) — number of processes to colocate with each identified process, or on the node of each identified process, respectively.
PMIX_SPAWN_TARGET (varies) — specify the allocation(s) to use when mapping the applications. The value is a char* PMIX_ALLOC_ID, or a pmix_data_array_t of such allocation-ID strings.
PMIX_APP_ARGV (char*) — the consolidated argv passed to the spawn command for the given application.
PMIX_INDEX_ARGV (bool) — mark each process’s argv with the rank of the process.
PMIX_WDIR_USER_SPECIFIED (bool) — indicate that the working directory was explicitly specified by the user (accompanies PMIX_WDIR).
PMIX_DISPLAY_MAP (bool) — display the resulting process placement map upon spawn.
PMIX_DISPLAY_MAP_DETAILED (bool) — display a highly detailed placement map upon spawn.
PMIX_STDIN_TGT (pmix_proc_t*) — the process that is to receive forwarded stdin.

Diagnostic and dry-run output:

PMIX_DISPLAY_ALLOCATION (bool) — display the resource allocation.
PMIX_DISPLAY_TOPOLOGY (char*) — comma-delimited list of hosts whose topology is to be displayed.
PMIX_DISPLAY_PROCESSORS (char*) — comma-delimited list of hosts whose available CPUs are to be displayed.
PMIX_DISPLAY_PARSEABLE_OUTPUT (bool) — render the requested display information in a format more amenable to machine parsing.
PMIX_REPORT_BINDINGS (bool) — report the binding of each individual process.
PMIX_REPORT_PHYSICAL_CPUS (bool) — report bindings using physical CPU IDs.
PMIX_SHOW_LAUNCH_PROGRESS (bool) — provide periodic progress reports on the job launch procedure (e.g., after every 100 processes have been spawned).
PMIX_DO_NOT_LAUNCH (bool) — execute all procedures to prepare the requested job for launch, but do not launch it. Typically combined with PMIX_DISPLAY_MAP or PMIX_DISPLAY_MAP_DETAILED for debugging.
PMIX_AGGREGATE_HELP (bool) — aggregate help messages, reporting each unique message once accompanied by the number of processes that reported it.

Environment control:

PMIX_SET_ENVAR / PMIX_ADD_ENVAR / PMIX_PREPEND_ENVAR / PMIX_APPEND_ENVAR (pmix_envar_t*) — set, add, prepend to, or append to an environment variable in the spawned processes’ environment.
PMIX_FIRST_ENVAR (pmix_envar_t*) — ensure the given value appears first in the specified environment variable, using the provided separator.
PMIX_UNSET_ENVAR (char*) — unset the named environment variable in the spawned processes’ environment, if present.
PMIX_ENVARS_HARVESTED (bool) — indicates that the requestor has already harvested and included the relevant environment variables.
PMIX_FWD_ENVIRONMENT (bool) — forward the local environment to each spawned process.
PMIX_SETUP_APP_ENVARS (bool) — direct the spawn library to harvest the environment variables relevant to the application’s programming model and include them with the request. May be placed in the job_info array or in an individual application’s info array. The full set of application-setup-data attributes is described under PMIx_server_setup_application(3).

Behavior, restart, and tool support:

PMIX_JOB_RECOVERABLE (bool) — the application supports recoverable operations.
PMIX_MAX_RESTARTS (uint32_t) — maximum number of times to restart a failed process.
PMIX_COSPAWN_APP (bool) — the designated application is to be spawned as a disconnected job.
PMIX_JOB_CONTINUOUS (bool) — the application is continuous; any failed processes should be immediately restarted.
PMIX_ABORT_NON_ZERO_TERM (bool) — abort the spawned job if any process terminates with non-zero status.
PMIX_SPAWN_CHILD_SEP (bool) — treat the spawned job as independent from the parent — i.e., do not terminate it if the parent terminates.
PMIX_REPORT_CHILD_SEP (bool) — report the exit status of any child jobs separately. If false, the reported status is zero when the primary and all child jobs exit normally, or the first non-zero status returned by any of them.
PMIX_RUNTIME_OPTIONS (char*) — environment-specific runtime directives that control job behavior.
PMIX_FORKEXEC_AGENT (char*) — command line of a fork/exec agent to be used for starting local processes.
PMIX_SPAWN_TOOL (bool) — the job being spawned is a tool.
PMIX_CMD_LINE (char*) — the command line executing in the specified namespace. This is informational — e.g., recorded for, or retrieved from, a running job — rather than a launch directive.

Debugger support:

These attributes support co-launching an application under debugger control and spawning debugger daemons alongside it.

PMIX_DEBUG_STOP_ON_EXEC (varies) — stop the specified rank(s) upon exec and notify that they are ready to be debugged. The value may be a bool (true for all ranks, false for none), a pmix_rank_t (a single rank, or PMIX_RANK_WILDCARD for all), or a pmix_data_array_t of individual processes.
PMIX_DEBUG_STOP_IN_INIT (varies) — stop the specified rank(s) inside PMIx_Init and notify that they are ready to be debugged. Same value forms as PMIX_DEBUG_STOP_ON_EXEC.
PMIX_DEBUG_STOP_IN_APP (varies) — direct the specified rank(s) to stop at an application-defined point and notify that they are ready to be debugged. Same value forms as PMIX_DEBUG_STOP_ON_EXEC.
PMIX_DEBUG_TARGET (pmix_proc_t*) — identifier of the process(es) to be debugged. A launcher spawning debugger daemons places this in the daemons’ job-level information so the daemons can self-determine their specific targets.
PMIX_DEBUG_DAEMONS_PER_PROC (uint16_t) — number of debugger daemons to spawn per application process.
PMIX_DEBUG_DAEMONS_PER_NODE (uint16_t) — number of debugger daemons to spawn on each node where the target job is executing.
PMIX_DEBUGGER_DAEMONS (bool) — the spawned application consists of debugger daemons.

When a process stopped by one of the PMIX_DEBUG_STOP_* directives is ready to be debugged, it generates a PMIX_READY_FOR_DEBUG event accompanied by the following OUT attribute (delivered to the debugger’s event handler, not passed to PMIx_Spawn):

PMIX_BREAKPOINT (char*) — string identifier of the breakpoint at which the process(es) are waiting.

Timeouts:

PMIX_TIMEOUT (int) — time, in seconds, before the operation should be declared to have timed out (0 means infinite).
PMIX_JOB_TIMEOUT (int) — time, in seconds, before the spawned job should time out.
PMIX_SPAWN_TIMEOUT (int) — time, in seconds, before the spawn operation itself should time out.
PMIX_TIMEOUT_STACKTRACES (bool) — include process stacktraces in the timeout report from a job.
PMIX_TIMEOUT_REPORT_STATE (bool) — report process states in the timeout report from a job.

Completion and termination notification:

PMIX_NOTIFY_COMPLETION (bool) — notify the parent process when the spawned job terminates, either normally or with an error.
PMIX_NOTIFY_PROC_TERMINATION (bool) — request that the launcher generate a PMIX_EVENT_PROC_TERMINATED event whenever any process in the spawned job terminates, normally or abnormally.
PMIX_NOTIFY_PROC_ABNORMAL_TERMINATION (bool) — request that the launcher generate a PMIX_EVENT_PROC_TERMINATED event only when a process terminates abnormally.
PMIX_NOTIFY_JOB_EVENTS (bool) — request that the launcher generate the PMIX_EVENT_JOB_START, PMIX_LAUNCH_COMPLETE, and PMIX_EVENT_JOB_END events, each including at least the job’s namespace and a PMIX_EVENT_TIMESTAMP.
PMIX_EVENT_SILENT_TERMINATION (bool) — suppress the event that is otherwise generated when this job terminates normally. See PMIx_Notify_event(3) for the notification-side description of this attribute.

Output forwarding:

These attributes control which of the spawned job’s streams are forwarded and how they are tagged, formatted, and directed. They supersede the older, now-deprecated non-IOF output attributes (see the note below).

Selecting which streams to forward:

PMIX_FWD_STDIN (bool) — forward this process’s stdin to the target processes.
PMIX_FWD_STDOUT (bool) — forward the spawned processes’ stdout to this process (typically used by a tool).
PMIX_FWD_STDERR (bool) — forward the spawned processes’ stderr to this process (typically used by a tool).
PMIX_FWD_STDDIAG (bool) — if a diagnostic channel exists, forward its output from the spawned processes to this process (typically used by a tool).

Formatting and destination of the forwarded output:

PMIX_IOF_TAG_OUTPUT (bool) — tag each line of output with the [local jobid,rank] of its source and the channel it came from.
PMIX_IOF_TAG_DETAILED_OUTPUT (bool) — tag output with the [local jobid,rank][hostname:pid] and channel of its source.
PMIX_IOF_TAG_FULLNAME_OUTPUT (bool) — tag output with the [nspace,rank] and channel of its source.
PMIX_IOF_RANK_OUTPUT (bool) — tag output with the rank it came from.
PMIX_IOF_TIMESTAMP_OUTPUT (bool) — timestamp each line of output.
PMIX_IOF_MERGE_STDERR_STDOUT (bool) — merge the stderr stream into the stdout stream of the application processes.
PMIX_IOF_XML_OUTPUT (bool) — format the forwarded output in XML.
PMIX_IOF_OUTPUT_RAW (bool) — do not buffer output into complete lines; emit characters as the stream delivers them.
PMIX_IOF_LOCAL_OUTPUT (bool) — write the output streams to the local stdout/stderr.
PMIX_IOF_OUTPUT_TO_FILE (char*) — direct application output into files of the form <filename>.rank, with both stdout and stderr redirected into it.
PMIX_IOF_OUTPUT_TO_DIRECTORY (char*) — direct application output into files of the form <directory>/<jobid>/rank.<rank>/stdout[err].
PMIX_IOF_FILE_PATTERN (bool) — treat the PMIX_IOF_OUTPUT_TO_FILE value as a pattern, suppressing the automatic annotation by nspace, rank, or other parameters.
PMIX_IOF_FILE_ONLY (bool) — output only into the designated files; do not also emit a copy to stdout/stderr.

Note

Several attributes historically used with PMIx_Spawn to control output forwarding — PMIX_TAG_OUTPUT, PMIX_TIMESTAMP_OUTPUT, PMIX_MERGE_STDERR_STDOUT, PMIX_OUTPUT_TO_FILE, and PMIX_OUTPUT_TO_DIRECTORY — are now deprecated in favor of the PMIX_IOF_* equivalents listed above under Output forwarding, as is PMIX_NON_PMI (indicating that the spawned processes will not call PMIx_Init); none of these should be used in new code.

Pseudo-terminal:

PMIX_SPAWN_PTY (bool) — spawn the processes under a pseudo-terminal.
PMIX_PTY_TERMIO (pmix_byte_object_t) — the contents of a termio structure describing the terminal settings for the pseudo-terminal.
PMIX_PTY_WSIZE (pmix_byte_object_t) — the contents of a window-size structure for the pseudo-terminal.

18.2.12.6. RETURN VALUE

For the blocking form, PMIX_SUCCESS indicates that the job was successfully launched and its namespace has been returned in nspace. For the non-blocking form, a return of PMIX_SUCCESS indicates only that the request was accepted for processing; the final status and assigned namespace are delivered to cbfunc.

PMIX_SUCCESS — the job was successfully launched.
PMIX_OPERATION_SUCCEEDED — (non-blocking form) the spawn completed atomically; cbfunc will not be called. In this case PMIx_Spawn returns the assigned namespace in nspace.
PMIX_ERR_JOB_ALLOC_FAILED — the job could not be executed due to failure to obtain the specified allocation.
PMIX_ERR_JOB_APP_NOT_EXECUTABLE — the specified executable could not be found or lacks execution privileges.
PMIX_ERR_JOB_NO_EXE_SPECIFIED — the request did not specify an executable.
PMIX_ERR_JOB_FAILED_TO_MAP — the launcher was unable to map the processes for the request.
PMIX_ERR_JOB_FAILED_TO_LAUNCH — one or more processes failed to launch.
PMIX_ERR_JOB_EXE_NOT_FOUND — the specified executable was not found.
PMIX_ERR_JOB_INSUFFICIENT_RESOURCES — insufficient resources to spawn the job.
PMIX_ERR_JOB_SYS_OP_FAILED — a system library operation failed.
PMIX_ERR_JOB_WDIR_NOT_FOUND — the specified working directory was not found.
PMIX_ERR_BAD_PARAM — an invalid argument was supplied — for example, an apps element supplying neither a cmd nor an argv.
PMIX_ERR_NOT_SUPPORTED — the PMIx server has no host environment capable of servicing the spawn request.
PMIX_ERR_UNREACH — the caller is a tool or client that is not connected to a PMIx server, and no local fork/exec fallback is available.
PMIX_ERR_NOT_AVAILABLE — the operation cannot be serviced because the library’s progress engine has been stopped.
PMIX_ERR_INIT — the PMIx library has not been initialized.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

18.2.12.7. NOTES

A launcher process (such as an intermediate launcher started by a tool) that is not connected to a PMIx server defaults to launching the applications via a local fork/exec, allowing tools to maintain a single code path for both the connected and disconnected cases. A tool or client that is not so privileged and cannot reach a server receives PMIX_ERR_UNREACH.

When PMIx_Spawn_nb returns PMIX_OPERATION_SUCCEEDED, the spawn was satisfied atomically and no callback will be made; callers must handle this status distinctly from PMIX_SUCCESS. The blocking PMIx_Spawn absorbs this case internally and simply returns PMIX_SUCCESS with the namespace populated.