11.2.1. OpenFabrics Interfaces (OFI) / Libfabric support
“OFI” stands for the OpenFabrics Interfaces, which are implemented in the libfabric
library; the two terms are typically used interchangeably.
Open MPI supports many different underlying networks via Libfabric, including (but not limited to):
AWS EFA
Cisco usNIC
Cornelis Networks Omni-Path
HPE Slingshot 11
In general, the OFI-based components in Open MPI auto-select themselves as appropriate at run time. The remaining sections describe the OFI-based components and the tuning options for specific OFI-based network types.
11.2.1.1. Libfabric (OFI) components in Open MPI
Open MPI has three main components for Libfabric (OFI) communication:
The
ofiMTL, available since Open MPI v1.10, is used with thecmPML for two-sided MPI communication (for example,MPI_SendandMPI_Recv). It requires that the Libfabric provider support reliable datagrams with ordered tagged messaging — specifically,FI_EP_RDMendpoints,FI_TAGGEDcapabilities, andFI_ORDER_SASordering.The
ofiBTL, available since Open MPI v4.0.0, is primarily intended for one-sided MPI communication (for example,MPI_Put), but can also support BTL send/receive operations. It requires that the Libfabric provider support reliable datagrams, RMA and atomic operations, and remote atomic completion notifications — specifically,FI_EP_RDMendpoints,FI_RMAandFI_ATOMICcapabilities, and theFI_DELIVERY_COMPLETEop flag.The
usnicBTL is used exclusively with Cisco usNIC-based networks. It auto-selects itself over the other OFI-based components when run on such networks.
Consult each Libfabric provider’s man page (for example,
fi_sockets(7)) to understand which provider will work for each of
the components above. Some providers must be paired with a Libfabric
utility provider; for example, the verbs provider needs the ofi_rxm
utility provider to supply reliable datagram endpoint support
(verbs;ofi_rxm).
Each component has MCA parameters that specify the Libfabric provider(s) to include in or exclude from the selection process. For example:
shell$ mpirun --mca pml cm --mca mtl ofi \
--mca mtl_ofi_provider_include psm2 ./mpi_hello
Each component also has its own component-specific parameters; use
ompi_info to list them. For example:
shell$ ompi_info --param mtl ofi --level 9
11.2.1.2. Point-to-point selection on EFA nodes with GPUs
Open MPI initializes every available PML and keeps the one that reports the
highest priority. The cm PML reports the priority of its MTL; the ofi
MTL reports 25 by default, and the ob1 PML reports 20. On most systems
cm with the ofi MTL is therefore selected wherever Libfabric is
available.
On nodes with GPUs whose MTL would use the efa provider, the ofi MTL
reports a priority of 10 instead, so that the ob1 PML with the ofi BTL
is selected. The ofi BTL can drive several network interfaces per process,
where the ofi MTL drives one, and it can move device buffers with RDMA.
The rule applies when all of the following hold:
the node has a GPU, as found from the PCI devices in the hwloc topology (a display controller from NVIDIA or AMD), which every process on the node sees the same way whatever
CUDA_VISIBLE_DEVICESis set to;the provider include and exclude settings allow
efa, and Libfabric offers it;the
mtl_ofi_priorityMCA parameter has not been set.
Nodes without GPUs and other providers keep the previous selection. To use
cm with the ofi MTL on such a node, select it explicitly, or set the
MTL’s priority, which disables the rule:
shell$ mpirun --mca pml cm ...
shell$ mpirun --mca mtl_ofi_priority 25 ...
The decision is reported with --mca opal_common_ofi_verbose 1. All
processes of a job must use the same PML: a job whose nodes differ (for
example, some with GPUs and some without) stops at startup with a PML mismatch
error, and needs --mca pml cm or --mca pml ob1 on the command line.
11.2.1.3. Provider and domain selection
Open MPI asks Libfabric for a list of fi_info structures that satisfy the
capabilities required by the selected component. An fi_info structure
describes a provider, its fabric, and a domain. In this context, a Libfabric
domain is the provider’s view of a usable network resource; on many systems it
corresponds to one network interface card or one provider-specific network
endpoint. The domain name is reported in the fi_info structure and is not
necessarily the same as the operating system device name.
The ofi MTL and the ofi BTL use the common Open MPI selection code.
The selection process is:
Open MPI applies the provider include or exclude setting to the structures returned by Libfabric. These settings select provider names, not arbitrary hardware device names.
Open MPI selects one compatible provider structure. When several structures describe domains of the same provider with the same requested capabilities, those structures are candidates for local-resource selection.
If the process is not bound to a set of processing units, Open MPI selects among the candidate domains using the process’s local rank. This is a round-robin choice intended to balance ranks across domains.
If the process is bound and the structures contain usable PCI information, Open MPI calculates device locality and selects a domain closest to the process. If several domains are equally close, the process’s rank within its processor package selects among the tied domains.
If accelerator locality is enabled and the accelerator and provider expose usable PCI information, Open MPI first attempts to select the domain closest to the accelerator. Ties are distributed using the local rank on that accelerator.
If locality information is unavailable, Open MPI falls back to the rank-based round-robin choice. A process that is not bound cannot receive a locality guarantee from the operating system.
This selection is performed independently by each MPI process. It does not
mean that every process opens every domain, and it does not guarantee that a
particular operating system interface is selected. The provider’s fi_info
structures, process binding, PCI topology information, and the selected Open MPI
component all affect the result. Open MPI then opens the selected Libfabric
fabric and domain and creates its endpoints and completion resources.
The common selection parameter is exposed through component-specific aliases. For example, these commands show the provider and selection diagnostics for the matching transport layer and the byte transfer layer:
shell$ mpirun --mca mtl_ofi_verbose 1 \
--mca mtl_ofi_provider_include cxi ./mpi_hello
shell$ mpirun --mca btl_ofi_verbose 1 \
--mca btl_ofi_provider_include cxi ./mpi_hello
The exact diagnostics and available component parameters depend on the Open
MPI and Libfabric versions in use. ompi_info is the authoritative way to
list the parameters in a particular installation.
11.2.1.4. Multiple network interfaces in the ofi BTL
When a node has more network interfaces than local processes, each process of
the ofi BTL opens several of them, one BTL module per interface, and the
ob1 PML spreads traffic over them. The interfaces are grouped by the PCI
root complex they sit under, and each process takes its interfaces from one
root complex: driving concurrent traffic through interfaces under more than one
root complex can be much slower than using fewer interfaces under one.
Contiguous blocks of local ranks share a root complex, which with the usual
placement of one rank per GPU puts each rank on the root complex of its GPU.
MCA parameter |
Default |
Meaning |
|---|---|---|
|
0 |
Number of interfaces each process opens; 0 divides them among the local processes. Experimental. |
|
65536 |
Payload bytes of one fragment, which is also the eager limit and the
largest send, clamped to the provider’s maximum message size. Device
buffers keep an eager limit of 4 KB unless
|
Messages up to the fragment size are sent eagerly, without the rendezvous round trip. The receiver copies an eager message once more than a rendezvous message, so a stream of messages of 16 KB to 32 KB to one peer can reach more bandwidth with a smaller fragment size, at the cost of latency.
11.2.1.5. LINKx provider
Libfabric also provides a provider named lnx (LINKx). LINKx can combine
multiple Libfabric providers or domains behind one tagged-message endpoint. It
is an optional Libfabric feature, not an Open MPI-specific network component.
Whether it is appropriate depends on the provider versions, the system
topology, and the performance goals of the deployment. HPE or the site
administrator should confirm that it is recommended for a particular
Slingshot installation before it is enabled.
The provider must be configured through the Libfabric environment variable
FI_LNX_PROV_LINKS. For example, the following requests one LINKx group
containing shared memory and two Slingshot domains:
shell$ export FI_LNX_PROV_LINKS="shm+cxi:cxi0,cxi1"
shell$ mpirun --mca pml cm --mca mtl ofi \
--mca mtl_ofi_provider_include lnx ./mpi_hello
The provider manual(man fi_lnx) describes additional forms, including multiple groups and provider-specific domain lists. All nodes in the MPI job must use compatible LINKx configurations and the same ordering of linked providers and domains. If shared memory is included, LINKx uses it for intra-node communication; other linked providers are used for off-node communication. LINKx can also distribute messages across multiple linked domains, according to its multi-rail selection policy.
There are important tradeoffs. The LINKx provider described by the current Libfabric manual supports tagged operations, but does not provide hardware offload such as hardware tag matching. Because memory registration does not identify the eventual operation or destination, LINKx registers memory with all linked providers. This can increase memory registration cost and may affect memory behavior. Measure an application with and without LINKx before making it a site-wide default.
11.2.1.6. HPE Slingshot
Open MPI supports HPE Slingshot through the Libfabric cxi provider. The
cxi provider maps Open MPI communication operations to the Slingshot
Cassini network interface card and Rosetta switch.
Open MPI normally selects its OpenFabrics Interfaces components automatically.
To request the tagged-message path explicitly, select the cm PML and
ofi MTL, and include the cxi provider. When the selected provider
supports hardware tag matching, message matching can be offloaded from the
host processor to the network interface card. The cxi provider supports
this capability. Tagged messages carry a numeric label that Open MPI uses to
match a send operation to the correct receive operation:
shell$ mpirun --mca pml cm --mca mtl ofi \
--mca mtl_ofi_provider_include cxi ./mpi_hello
Use the Libfabric fi_info utility to confirm that the installed Libfabric
can find the cxi provider:
shell$ fi_info -p cxi
Important
When using the HPE CXI provider with mpirun as the
job launcher, it is recommended to set the PRRTE
ras_base_launch_orted_on_hn MCA parameter to 1 by
adding --prtemca ras_base_launch_orted_on_hn 1 to
the job launch command line. This ensures that MPI
processes launched on the first node of an allocation
are able to use the CXI provider.
For more information, refer to the Libfabric web site.
11.2.1.7. Omni-Path: multi-rail with multiple HFI cards
The multi-rail feature lets a single process use multiple Host Fabric Interface (HFI) cards to transfer a message, improving message bandwidth. The PSM2 library provides multi-rail support, which is off by default. The behavior is controlled with the following environment variables:
PSM2_MULTIRAIL=[0,1,2]: 0 disables multi-rail, 1 enables it across all HFIs in the system, and 2 enables multi-rail within a NUMA node.PSM2_MULTIRAIL_MAP=unit:port,unit:port,...
These variables may be set on the mpirun command line or in the
environment. For example:
shell$ mpirun --mca mtl [psm2|ofi] -x PSM2_MULTIRAIL=1 \
-n 2 -H host1,host2 ./a.out
Note
When using the OFI MTL, ensure that the PSM2 OFI provider is used for communication with OPA devices.
11.2.1.8. Omni-Path: multi-HFI support in PSM2
Multi-HFI support describes the use of multiple HFIs in a system among the MPI ranks local to a node, in order to load-balance the hardware resources. It differs from the multi-rail feature, which allows a single process to use all HFIs in the system. For an MPI job with multiple processes on a single node, the default PSM2 behavior depends on the affinity settings of the MPI process: the PSM2 library defaults to using the HFI that is in the same NUMA node as the MPI process.
Users can restrict a process to a single HFI with an environment variable:
HFI_UNIT=N: valid values ofNare 0, 1, 2, and 3.
More details can be found in the PSM2 Programmer’s Guide and the Omni-Path Fabric Performance Tuning Guide; see the Cornelis Networks Customer Center.
11.2.1.9. GPU memory and Libfabric providers
When an MPI buffer resides in graphics processing unit memory, Open MPI can
request Libfabric support for heterogeneous memory through the FI_HMEM
capability. The ofi matching transport layer and byte transfer layer
request this capability by default when it is available in the build. The
matching transport layer and byte transfer layer can disable the request
with mtl_ofi_disable_hmem and btl_ofi_disable_hmem, respectively.
Requesting heterogeneous-memory support does not guarantee direct network access to every type of GPU memory. The selected provider must advertise the required capability, and the Libfabric build must include support for the GPU runtime in use. If no suitable provider is found with heterogeneous-memory requirements, Open MPI may retry provider discovery without that requirement; this can result in host-memory staging or a provider that does not support direct GPU buffers.
LINKx deserves additional care for GPU buffers. LINKx forwards memory registration to all providers in a link, and the linked providers may have different GPU-memory capabilities. A LINKx configuration should therefore contain only providers that support the intended buffer type and memory registration mode. Validate the configuration with a representative application and consult the installed provider manuals before using GPU buffers in production.
11.2.1.10. Memory monitors
Open MPI and Libfabric both cache memory registrations. Each cache needs a memory monitor that reports when cached memory is unmapped or remapped.
Open MPI’s monitor comes from the memory framework, which has three
components:
ucmreports the releases that UCX’s memory event layer (UCM) sees. It is only used when UCM has already hooked the C library’s memory functions, for example because a UCX component is loaded.kdreg2uses the kdreg2 kernel module, which watches each registered region. It is built whenconfigurefindskdreg2.h(see--with-kdreg2).patcherinterceptsmunmap(2)and related calls.
Open MPI uses the highest-priority component that it can open (see
the memory_ucm_priority, memory_kdreg2_priority and
memory_patcher_priority MCA parameters), and moves on to the next
one when a component cannot be used, such as kdreg2 when its
device is missing. Set the memory MCA parameter to choose one;
for example, --mca memory kdreg2. If none of the named components
can be used, Open MPI prints an error and runs without memory hooks.
The component that Open MPI uses also decides Libfabric’s monitor, as follows:
With
ucmorpatcher, Open MPI exports its monitor to Libfabric (version 1.14 and later), and Libfabric uses it instead of the monitor thatFI_MR_CACHE_MONITORnames.With
kdreg2, Open MPI does not export a monitor, becausekdreg2only watches the regions that Open MPI registers. Libfabric picks its own monitor, whichFI_MR_CACHE_MONITORcontrols. When built with kdreg2 support, Libfabric 2.4 and later uses its ownkdreg2monitor by default; Open MPI and Libfabric then each open the kdreg2 device and watch their own regions.
With OMPI_MCA_opal_common_ofi_verbose=1, Open MPI prints a message
when it does not export its monitor.