18.2.23. PMIx_Process_monitor

PMIx_Process_monitor, PMIx_Process_monitor_nb — Request that processes, files, or resources be monitored for activity or usage.

18.2.23.1. SYNOPSIS

#include <pmix.h>

pmix_status_t PMIx_Process_monitor(const pmix_info_t *monitor,
                                   pmix_status_t error,
                                   const pmix_info_t directives[], size_t ndirs,
                                   pmix_info_t **results, size_t *nresults);

pmix_status_t PMIx_Process_monitor_nb(const pmix_info_t *monitor,
                                      pmix_status_t error,
                                      const pmix_info_t directives[], size_t ndirs,
                                      pmix_info_cbfunc_t cbfunc, void *cbdata);

18.2.23.1.1. Python Syntax

from pmix import *

foo = PMIxClient()
# ... after a successful foo.init() ...
# the monitor action is a single Python ``pmix_info_t`` dictionary
monitor = {'key': PMIX_MONITOR_HEARTBEAT,
           'value': {'value': True, 'val_type': PMIX_BOOL}}
# the directives are a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_MONITOR_HEARTBEAT_TIME,
           'value': {'value': 5, 'val_type': PMIX_UINT32}}]
rc, results = foo.monitor(monitor, PMIX_MONITOR_HEARTBEAT_ALERT, pydirs)

18.2.23.2. INPUT PARAMETERS

monitor: Pointer to a single pmix_info_t(5) structure whose key specifies the monitoring action being requested — for example PMIX_MONITOR_HEARTBEAT to have the server watch the requestor for periodic heartbeats, or PMIX_MONITOR_FILE to monitor a file for signs of life. A NULL value returns PMIX_ERR_BAD_PARAM.
error: The pmix_status_t code the monitor is to use when generating an event notification alerting that the monitored condition has been triggered (or to carry a periodic resource-usage update). The default range of the resulting event is PMIX_RANGE_NAMESPACE; this can be changed with a PMIX_RANGE directive.
directives: Pointer to an array of pmix_info_t(5) structures characterizing the request — e.g., the checking frequency or the set of target processes, nodes, or files (see DIRECTIVES). A NULL value is supported when no directives are desired.
ndirs: Number of elements in the directives array.

18.2.23.3. OUTPUT PARAMETERS

results (blocking form): Address where a pointer to an array of pmix_info_t(5) structures containing the results of the request is returned. Initialized to NULL on entry; any returned array must be released by the caller with PMIX_INFO_FREE.
nresults (blocking form): Address where the number of elements in the results array is returned.

The non-blocking form replaces results/nresults with a callback:

cbfunc: Callback function of type pmix_info_cbfunc_t invoked with the final status and any returned data once the request has been processed.
cbdata: Opaque pointer that is passed, unmodified, to cbfunc.

18.2.23.4. DESCRIPTION

Request that application processes, files, and/or resources be monitored. PMIx_Process_monitor is the blocking form: it does not return until the request has been processed, at which point any measured values are returned in the results array. PMIx_Process_monitor_nb is the non-blocking form: it returns immediately, and the provided cbfunc is invoked with the final status and any returned data.

This API serves two purposes:

Liveness monitoring. Request that the server watch a process or file for activity — for example, that it monitor a given process for periodic heartbeats as an indication that the process has not become “wedged”. When a monitor detects the specified alarm condition, it generates an event notification using the provided error code and passes along any relevant information. It is up to the caller to register a corresponding event handler (e.g., for PMIX_MONITOR_HEARTBEAT_ALERT or PMIX_MONITOR_FILE_ALERT).
Resource-usage reporting. Report resource-usage statistics for processes, nodes, disks, or network interfaces. This can be done on a per-request basis, or updated periodically on a time interval specified by the PMIX_MONITOR_RESOURCE_RATE directive.

The monitor argument is an attribute naming the type of monitor being requested. The directives argument characterizes the request (e.g., which file to monitor, or the checking interval and number of allowed misses).

A process that is being watched for heartbeats emits them with the companion PMIx_Heartbeat(3) function; a heartbeat may also be sent by passing the PMIX_SEND_HEARTBEAT attribute as the monitor action.

As with all non-blocking PMIx APIs, callers of PMIx_Process_monitor_nb must keep the monitor and directives arrays valid until cbfunc is invoked.

18.2.23.5. DIRECTIVES

The monitor argument may carry any of the following actions:

PMIX_MONITOR_HEARTBEAT (bool) — register to have the server monitor the requestor for heartbeats.
PMIX_SEND_HEARTBEAT (bool) — send a heartbeat to the local server (equivalent to calling PMIx_Heartbeat(3)). A server is not permitted to use this action and receives PMIX_ERR_BAD_PARAM.
PMIX_MONITOR_FILE (char*) — register to monitor the named file for signs of life.
PMIX_MONITOR_PROC_RESOURCE_USAGE / PMIX_MONITOR_NODE_RESOURCE_USAGE / PMIX_MONITOR_DISK_RESOURCE_USAGE / PMIX_MONITOR_NET_RESOURCE_USAGE (pmix_data_array_t*) — report usage of the resources specified in the provided array for processes, nodes, disks, or network interfaces respectively.
PMIX_MONITOR_CANCEL (char*) — cancel a previously requested monitoring action, identified by the string supplied via PMIX_MONITOR_ID (NULL cancels all).

Action-specific directives (placed in the directives array) include:

PMIX_MONITOR_HEARTBEAT_TIME (uint32_t) — time, in seconds, before a missed heartbeat is declared.
PMIX_MONITOR_HEARTBEAT_DROPS (uint32_t) — number of heartbeats that may be missed before the alarm is raised.
PMIX_MONITOR_FILE_SIZE (bool) — track whether the file’s size is growing to determine that the application is running.
PMIX_MONITOR_FILE_ACCESS (char*) — track time since the file was last accessed.
PMIX_MONITOR_FILE_MODIFY (char*) — track time since the file was last modified.
PMIX_MONITOR_FILE_CHECK_TIME (uint32_t) — time, in seconds, between file checks.
PMIX_MONITOR_FILE_DROPS (uint32_t) — number of file checks that may be missed before the alarm is raised.
PMIX_MONITOR_RESOURCE_RATE (uint32_t) — report resource usage every N seconds.
PMIX_MONITOR_TARGET_PROCS (pmix_data_array_t*) — array of process IDs identifying the processes to be monitored.
PMIX_MONITOR_TARGET_PIDS (pmix_data_array_t*) — array of pmix_node_pid_t structures to be monitored.
PMIX_MONITOR_TARGET_NODES (pmix_data_array_t*) — array of string host names to be monitored.
PMIX_MONITOR_TARGET_NODEIDS (pmix_data_array_t*) — array of uint32_t node IDs to be monitored.
PMIX_MONITOR_TARGET_DISKS (pmix_data_array_t*) — array of disk identifiers to be monitored.
PMIX_MONITOR_TARGET_NETS (pmix_data_array_t*) — array of network identifiers to be monitored.

General directives that may accompany any action include:

PMIX_MONITOR_ID (char*) — string identifier for this request, used later to cancel it.
PMIX_MONITOR_APP_CONTROL (bool) — the application wishes to control the response when the monitor is triggered.
PMIX_MONITOR_LOCAL_ONLY (bool) — restrict data collection to the local host, regardless of any provided targets.
PMIX_MONITOR_PROXY (pmix_proc_t*) — the process on whose behalf the monitoring is being requested, if different from the caller.
PMIX_RANGE (pmix_data_range_t) — non-default range to use when generating the associated event for this monitoring action.

When a monitoring request involves other nodes, the library adds the PMIX_USERID and PMIX_GRPID of the requesting process to the directives it passes to its host environment.

18.2.23.6. RESOURCE USAGE ATTRIBUTES

The following attributes describe the individual resource-usage statistics associated with the PMIX_MONITOR_PROC_RESOURCE_USAGE, PMIX_MONITOR_NODE_RESOURCE_USAGE, PMIX_MONITOR_DISK_RESOURCE_USAGE, and PMIX_MONITOR_NET_RESOURCE_USAGE actions. They serve a dual purpose: placed (as keys, with their values ignored) in the array accompanying a *_RESOURCE_USAGE action, they select which specific statistics are to be reported; they then label the corresponding values in the returned data. If the accompanying array is NULL, all available statistics are reported. The availability of any given statistic varies across implementations and operating systems. Each per-sample container places its subject identifier in the first element of the array; the ordering of the remaining elements is arbitrary.

Process resource usage:

PMIX_PROC_RESOURCE_USAGE (pmix_data_array_t*) — array of pmix_info_t(5) describing the resource usage of a process, with the process ID (marked by PMIX_PROCID) as the first element.
PMIX_PROC_OS_STATE (char*) — the state of the process as reported by the OS (on Linux, a single character).
PMIX_PROC_TIME (struct timeval) — cumulative CPU time.
PMIX_PROC_PERCENT_CPU (float) — percent CPU utilization by the process (typically the CPU-time / real-time ratio, as a percentage).
PMIX_PROC_PRIORITY (int32_t) — priority of the process (higher numbers mean higher priority).
PMIX_PROC_NUM_THREADS (uint16_t) — number of threads operating in the process.
PMIX_PROC_PSS (float) — proportional share size — the non-swapped physical memory, with shared memory proportionally accounted to all tasks mapping it (MBytes).
PMIX_PROC_VSIZE (float) — virtual memory size of the process (MBytes).
PMIX_PROC_RSS (float) — resident set size — the non-swapped physical memory the task has used (MBytes).
PMIX_PROC_PEAK_VSIZE (float) — peak virtual memory size of the process (MBytes).
PMIX_PROC_CPU (uint16_t) — the processor on which the process last executed.
PMIX_PROC_SAMPLE_TIME (time_t) — time when the sample was taken. Always included in returned process-usage data.

Disk resource usage:

PMIX_DISK_RESOURCE_USAGE (pmix_data_array_t*) — array of pmix_info_t(5) describing the resource usage of a disk, with the disk name (marked by PMIX_DISK_ID) as the first element.
PMIX_DISK_ID (char*) — string identifier of a disk.
PMIX_DISK_READ_COMPLETED (uint64_t) — number of completed read operations.
PMIX_DISK_READ_MERGED (uint64_t) — number of merged reads.
PMIX_DISK_READ_SECTORS (uint64_t) — number of sectors read.
PMIX_DISK_READ_MILLISEC (uint64_t) — milliseconds spent reading the disk.
PMIX_DISK_WRITE_COMPLETED (uint64_t) — number of completed write operations.
PMIX_DISK_WRITE_MERGED (uint64_t) — number of merged writes.
PMIX_DISK_WRITE_SECTORS (uint64_t) — number of sectors written.
PMIX_DISK_WRITE_MILLISEC (uint64_t) — milliseconds spent writing to the disk.
PMIX_DISK_IO_IN_PROGRESS (uint64_t) — number of disk I/O operations in progress.
PMIX_DISK_IO_MILLISEC (uint64_t) — milliseconds spent in I/O operations.
PMIX_DISK_IO_WEIGHTED (uint64_t) — number of I/Os in progress times the milliseconds spent doing I/O since the last update — an indicator of accumulating backlog.
PMIX_DISK_SAMPLE_TIME (time_t) — time when the sample was taken. Always included in returned disk-usage data.

Network resource usage:

PMIX_NETWORK_RESOURCE_USAGE (pmix_data_array_t*) — array of pmix_info_t(5) describing the resource usage of a network interface, with the interface name (marked by PMIX_NETWORK_ID) as the first element.
PMIX_NETWORK_ID (char*) — string identifier of a network interface.
PMIX_NET_RECVD_BYTES (uint64_t) — number of bytes received.
PMIX_NET_RECVD_PCKTS (uint64_t) — number of packets received.
PMIX_NET_RECVD_ERRS (uint64_t) — number of receive errors.
PMIX_NET_SENT_BYTES (uint64_t) — number of bytes sent.
PMIX_NET_SENT_PCKTS (uint64_t) — number of packets sent.
PMIX_NET_SENT_ERRS (uint64_t) — number of send errors.
PMIX_NET_SAMPLE_TIME (time_t) — time when the sample was taken. Always included in returned network-usage data.

Node resource usage:

PMIX_NODE_RESOURCE_USAGE (pmix_data_array_t*) — array of pmix_info_t(5) describing the resource usage of a node, with the node ID (marked by PMIX_HOSTNAME or PMIX_NODEID) as the first element.
PMIX_NODE_LOAD_AVG (float) — load average over the last minute.
PMIX_NODE_LOAD_AVG5 (float) — load average over the last five minutes.
PMIX_NODE_LOAD_AVG15 (float) — load average over the last fifteen minutes.
PMIX_NODE_MEM_TOTAL (float) — total usable RAM — physical RAM minus reserved bits and kernel binary code (MBytes).
PMIX_NODE_MEM_FREE (float) — total free RAM (MBytes).
PMIX_NODE_MEM_BUFFERS (float) — temporary storage for raw disk blocks (MBytes).
PMIX_NODE_MEM_CACHED (float) — in-memory cache for files read from disk, plus tmpfs and shmem (excluding PMIX_NODE_MEM_SWAP_CACHED) (MBytes).
PMIX_NODE_MEM_SWAP_CACHED (float) — memory that was swapped out and swapped back in but is still also present in the swapfile (MBytes).
PMIX_NODE_SWAP_TOTAL (float) — total amount of swap space available (MBytes).
PMIX_NODE_MEM_SWAP_FREE (float) — memory evicted from RAM and temporarily on disk (MBytes).
PMIX_NODE_MEM_MAPPED (float) — files that have been mmapped, such as libraries (MBytes).
PMIX_NODE_SAMPLE_TIME (time_t) — time when the sample was taken. Always included in returned node-usage data.

18.2.23.7. RETURN VALUE

For the blocking form, PMIX_SUCCESS indicates that the request was processed without error and that any results have been placed in the results array. For the non-blocking form, a return of PMIX_SUCCESS indicates only that the request was accepted for processing; the final status and any data are delivered to cbfunc.

PMIX_SUCCESS — the request was processed successfully.
PMIX_ERR_BAD_PARAM — an invalid argument was supplied — for example, a NULL monitor, or a server attempting to use PMIX_SEND_HEARTBEAT.
PMIX_ERR_NOT_SUPPORTED — the request involves other nodes but the host environment provides no monitoring support.
PMIX_ERR_UNREACH — the caller is not a server and its local PMIx server could not be reached.
PMIX_ERR_NOT_AVAILABLE — the operation cannot be serviced because the library’s progress engine has been stopped.
PMIX_ERR_INIT — the PMIx library has not been initialized.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

18.2.23.8. NOTES

A PMIX_SUCCESS return only indicates that the request was executed without error; it does not guarantee a non-NULL results array when requesting resource-usage statistics. For example, a request for the resource usage of processes on a node that is not currently running any user-level processes returns success with an empty results array because no statistics were produced.

Support for individual monitoring actions and directives is implementation- and host-environment-dependent.