20.2.38. PMIx_Compute_distances

PMIx_Compute_distances, PMIx_Compute_distances_nb — Compute the distances from a specified process location, or from a specified device, to the local devices.

20.2.38.1. SYNOPSIS

#include <pmix.h>

pmix_status_t PMIx_Compute_distances(pmix_topology_t *topo,
                                     pmix_cpuset_t *cpuset,
                                     pmix_info_t info[], size_t ninfo,
                                     pmix_device_distance_t *distances[],
                                     size_t *ndist);

pmix_status_t PMIx_Compute_distances_nb(pmix_topology_t *topo,
                                        pmix_cpuset_t *cpuset,
                                        pmix_info_t info[], size_t ninfo,
                                        pmix_device_dist_cbfunc_t cbfunc,
                                        void *cbdata);

20.2.38.1.1. Python Syntax

from pmix import *

foo = PMIxClient()
# ... after a successful foo.init() ...
# load the local topology first
rc = foo.load_topology()
# the cpuset is a Python dictionary describing the process location
pycpus = {'source': "hwloc", 'cpus': [0, 1]}
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_DEVICE_TYPE,
           'value': PMIX_DEVTYPE_NETWORK, 'val_type': PMIX_UINT64}]
rc, distances = foo.compute_distances(pycpus, pydirs)

# the non-blocking form returns as soon as the request has been accepted
# and reports the result by executing a callback on the PMIx progress
# thread. The callback is run if and only if the call returned
# PMIX_SUCCESS, and must not itself make a blocking PMIx call.
def distcb(status, distances, cbdata):
    # distances is a list of dictionaries, each carrying uuid, osname,
    # type, mindist and maxdist
    print("distances:", foo.error_string(status), distances)
rc = foo.compute_distances_nb(pycpus, pydirs, distcb, "mycbdata")

20.2.38.2. INPUT PARAMETERS

topo: Pointer to a pmix_topology_t describing the topology of the node where the process is located. A NULL value indicates that the topology of the local node is to be used. If the caller’s own topology has not yet been loaded, the library will attempt to load it (see PMIx_Load_topology(3)) or, failing that, relay the request to the local PMIx server.
cpuset: Pointer to a pmix_cpuset_t identifying the location (the set of processing units to which the process is bound) from which distances are to be computed. A NULL value indicates that the caller’s own location is to be used; if it has not yet been determined, the library will attempt to obtain it or relay the request to the local PMIx server. Ignored, and may be NULL, when the PMIX_DEVICE_DIST_ORIGIN directive is given.
info: Pointer to an array of pmix_info_t(5) structures describing the device type(s) or specific device(s) whose distance is to be computed (see DIRECTIVES). A NULL value (with ninfo of zero) requests distances to all device types supported by the underlying topology description.
ninfo: Number of elements in the info array.

For PMIx_Compute_distances_nb(3):

cbfunc: Callback function of type pmix_device_dist_cbfunc_t to be invoked when the operation is complete.
cbdata: Opaque data pointer to be passed to cbfunc when invoked.

20.2.38.3. OUTPUT PARAMETERS

For the blocking form PMIx_Compute_distances(3):

distances: On successful return, *distances points to a newly allocated array of pmix_device_distance_t structures containing the minimum and maximum distances from the specified process location to each of the requested devices. Each element reports the uuid, osname, type, mindist, and maxdist of a device. The array is allocated by the library and ownership passes to the caller, who is responsible for releasing it (for example, with PMIx_Device_distance_free()).
ndist: On successful return, *ndist holds the number of elements in the returned distances array.

On entry, the blocking form initializes *distances to NULL and *ndist to zero.

For the non-blocking form, the array of pmix_device_distance_t and its count are delivered as the dist and ndist arguments of the callback function. The data provided to the callback is owned by the library; the callback is passed a release_fn (and release_cbdata) that must be invoked when the caller is finished with the dist array.

20.2.38.4. DESCRIPTION

Compute the distances between a process at a given location and the devices available on a node. Both the minimum and maximum distance fields in each element of the returned array are filled with the respective distances between the process location and the device types (or specific devices) identified by the info directives. In the absence of directives, distances to the network, OpenFabrics, GPU and coprocessor devices on the node are returned.

Alternatively, distances can be computed from a device rather than from a process location by naming that device with the PMIX_DEVICE_DIST_ORIGIN directive — for example, to find the network devices nearest the GPU a process is using. The cpuset plays no part in that case, so a process need not be bound to ask. The origin device itself is included in the result, at a distance of zero, if it is of a requested type.

Distances are relative, and smaller means closer. A distance from a process location is governed first by where the device attaches to the processor hierarchy (the package or NUMA domain it is local to); among devices that attach at the same place, a device fewer PCIe levels (bridges and switches) below that point is closer. A distance from a device follows the path between the two devices: devices under the same PCIe switch are closer than devices under the same host bridge, which are closer than devices on different host bridges of the same package, which are closer than devices on different packages. Because every device under a package is equally close to that package’s processors, only a distance measured from a device can distinguish the host bridges within a package.

PMIx_Compute_distances(3) is the blocking form: it does not return until the computation completes, at which point the result array is returned through the distances and ndist parameters.

PMIx_Compute_distances_nb(3) is the non-blocking form: it returns immediately (PMIX_SUCCESS indicating that the request was successfully initiated) and delivers the result later by invoking cbfunc. As with all non-blocking PMIx operations, the caller must ensure that the topo, cpuset, and info arguments remain valid until cbfunc has been invoked.

If the local PMIx library can satisfy the request itself (i.e., it has access to a suitable topology and cpuset), it does so directly. Otherwise, a connected client or tool relays the request to its local PMIx server. A process acting solely as a server, or a process that is not connected to a server and cannot compute the distances locally, cannot service the request.

20.2.38.5. DIRECTIVES

The following attributes are optional for PMIx implementations and may be passed in the info array to select the devices whose distances are to be computed:

PMIX_DEVICE_DISTANCES ("pmix.dev.dist") — request the return of an array of pmix_device_distance_t describing the distances to devices on the local node.
PMIX_DEVICE_TYPE ("pmix.dev.type") — a pmix_device_type_t bitmask specifying the type(s) of device whose distances are to be computed. Values include PMIX_DEVTYPE_BLOCK, PMIX_DEVTYPE_GPU, PMIX_DEVTYPE_NETWORK, PMIX_DEVTYPE_OPENFABRICS, PMIX_DEVTYPE_DMA, PMIX_DEVTYPE_COPROC, and others defined in pmix_common.h.
PMIX_DEVICE_ID ("pmix.dev.id") — a system-wide UUID or node-local OS name (char*) identifying a particular device. A device may also be named by its vendor identity (e.g., an NVIDIA GPU- UUID) or its PCI bus id ("0000:06:00.0"). The attribute may be given more than once to request distances to several devices.
PMIX_DEVICE_DIST_ORIGIN ("pmix.dev.dist.origin") — a char* naming the device from which distances are to be computed, in any of the forms accepted by PMIX_DEVICE_ID, in place of a process location. The cpuset argument is ignored when this directive is given. Returns PMIX_ERR_NOT_FOUND if no device answers to the name, and PMIX_ERR_BAD_PARAM if more than one does.

20.2.38.6. RETURN VALUE

Returns PMIX_SUCCESS on success. On error, a negative value corresponding to a PMIx error constant is returned, including:

PMIX_ERR_INIT — the PMIx library has not been initialized.
PMIX_ERR_NOT_AVAILABLE — the operation cannot be serviced because the library’s progress engine has been stopped.
PMIX_ERR_UNREACH — the request could not be satisfied locally and the local PMIx server could not be reached (or the caller is a server, or is not connected to a server).
PMIX_ERR_NOT_SUPPORTED — PMIX_DEVICE_DIST_ORIGIN was given, the request could not be satisfied locally, and the local PMIx server predates support for that directive.

For the blocking form, the value returned is the status of the completed operation. For the non-blocking form, a return of PMIX_SUCCESS indicates only that the request was successfully initiated; the final status of the operation is reported as the status argument to cbfunc. Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

20.2.38.7. NOTES

The PMIX_DEVICE_DIST_CREATE and PMIX_DEVICE_DIST_FREE convenience macros are deprecated. New code should manage pmix_device_distance_t arrays using the PMIx_Device_distance_create() and PMIx_Device_distance_free() functions.

A process whose threads are not all bound to the same location may return inconsistent results from calls to this API by different threads if the PMIX_CPUBIND_THREAD binding envelope was used when generating the cpuset.

20.2.38.8. PROGRESS THREAD RESTRICTION

A blocking PMIx call must not be made from within the PMIx progress thread. Any code the library itself invokes runs on that thread: an event handler registered through PMIx_Register_event_handler(3), a callback passed to a non-blocking PMIx API, and — in a server or tool — the completion of a host-module up-call. A blocking call waits for work that the progress thread has to perform, so making one from that thread waits for itself and never returns. The PMIx Standard disallows it, and there is no way for an implementation to service such a request.

Where this call has a blocking form — including the blocking behavior a non-blocking entry point adopts when it is passed a NULL cbfunc — that form detects the situation and returns PMIX_ERR_WOULD_BLOCK immediately, accompanied by a diagnostic naming the call. Nothing is done and no callback is invoked.

PMIX_ERR_WOULD_BLOCK here is not a transient condition to retry: it reports a call that cannot be serviced from where it was made. Reissue it as the non-blocking form with a callback, or from a thread of your own.