11.2.1. OpenFabrics Interfaces (OFI) / Libfabric support

“OFI” stands for the OpenFabrics Interfaces, which are implemented in the libfabric library; the two terms are typically used interchangeably.

Open MPI supports many different underlying networks via Libfabric, including (but not limited to):

  • AWS EFA

  • Cisco usNIC

  • Cornelis Networks Omni-Path

  • HPE Slingshot 11

In general, the OFI-based components in Open MPI auto-select themselves as appropriate at run time. The remaining sections describe the OFI-based components and the tuning options for specific OFI-based network types.

11.2.1.1. Libfabric (OFI) components in Open MPI

Open MPI has three main components for Libfabric (OFI) communication:

  1. The ofi MTL, available since Open MPI v1.10, is used with the cm PML for two-sided MPI communication (for example, MPI_Send and MPI_Recv). It requires that the Libfabric provider support reliable datagrams with ordered tagged messaging — specifically, FI_EP_RDM endpoints, FI_TAGGED capabilities, and FI_ORDER_SAS ordering.

  2. The ofi BTL, available since Open MPI v4.0.0, is primarily intended for one-sided MPI communication (for example, MPI_Put), but can also support BTL send/receive operations. It requires that the Libfabric provider support reliable datagrams, RMA and atomic operations, and remote atomic completion notifications — specifically, FI_EP_RDM endpoints, FI_RMA and FI_ATOMIC capabilities, and the FI_DELIVERY_COMPLETE op flag.

  3. The usnic BTL is used exclusively with Cisco usNIC-based networks. It auto-selects itself over the other OFI-based components when run on such networks.

Consult each Libfabric provider’s man page (for example, fi_sockets(7)) to understand which provider will work for each of the components above. Some providers must be paired with a Libfabric utility provider; for example, the verbs provider needs the ofi_rxm utility provider to supply reliable datagram endpoint support (verbs;ofi_rxm).

Each component has MCA parameters that specify the Libfabric provider(s) to include in or exclude from the selection process. For example:

shell$ mpirun --mca pml cm --mca mtl ofi \
       --mca mtl_ofi_provider_include psm2 ./mpi_hello

Each component also has its own component-specific parameters; use ompi_info to list them. For example:

shell$ ompi_info --param mtl ofi --level 9

11.2.1.2. Point-to-point selection on EFA nodes with GPUs

Open MPI initializes every available PML and keeps the one that reports the highest priority. The cm PML reports the priority of its MTL; the ofi MTL reports 25 by default, and the ob1 PML reports 20. On most systems cm with the ofi MTL is therefore selected wherever Libfabric is available.

On nodes with GPUs whose MTL would use the efa provider, the ofi MTL reports a priority of 10 instead, so that the ob1 PML with the ofi BTL is selected. The ofi BTL can drive several network interfaces per process, where the ofi MTL drives one, and it can move device buffers with RDMA. The rule applies when all of the following hold:

  • the node has a GPU, as found from the PCI devices in the hwloc topology (a display controller from NVIDIA or AMD), which every process on the node sees the same way whatever CUDA_VISIBLE_DEVICES is set to;

  • the provider include and exclude settings allow efa, and Libfabric offers it;

  • the mtl_ofi_priority MCA parameter has not been set.

Nodes without GPUs and other providers keep the previous selection. To use cm with the ofi MTL on such a node, select it explicitly, or set the MTL’s priority, which disables the rule:

shell$ mpirun --mca pml cm ...
shell$ mpirun --mca mtl_ofi_priority 25 ...

The decision is reported with --mca opal_common_ofi_verbose 1. All processes of a job must use the same PML: a job whose nodes differ (for example, some with GPUs and some without) stops at startup with a PML mismatch error, and needs --mca pml cm or --mca pml ob1 on the command line.

11.2.1.3. Provider and domain selection

Open MPI asks Libfabric for a list of fi_info structures that satisfy the capabilities required by the selected component. An fi_info structure describes a provider, its fabric, and a domain. In this context, a Libfabric domain is the provider’s view of a usable network resource; on many systems it corresponds to one network interface card or one provider-specific network endpoint. The domain name is reported in the fi_info structure and is not necessarily the same as the operating system device name.

The ofi MTL and the ofi BTL use the common Open MPI selection code. The selection process is:

  1. Open MPI applies the provider include or exclude setting to the structures returned by Libfabric. These settings select provider names, not arbitrary hardware device names.

  2. Open MPI selects one compatible provider structure. When several structures describe domains of the same provider with the same requested capabilities, those structures are candidates for local-resource selection.

  3. If the process is not bound to a set of processing units, Open MPI selects among the candidate domains using the process’s local rank. This is a round-robin choice intended to balance ranks across domains.

  4. If the process is bound and the structures contain usable PCI information, Open MPI calculates device locality and selects a domain closest to the process. If several domains are equally close, the process’s rank within its processor package selects among the tied domains.

  5. If accelerator locality is enabled and the accelerator and provider expose usable PCI information, Open MPI first attempts to select the domain closest to the accelerator. Ties are distributed using the local rank on that accelerator.

  6. If locality information is unavailable, Open MPI falls back to the rank-based round-robin choice. A process that is not bound cannot receive a locality guarantee from the operating system.

This selection is performed independently by each MPI process. It does not mean that every process opens every domain, and it does not guarantee that a particular operating system interface is selected. The provider’s fi_info structures, process binding, PCI topology information, and the selected Open MPI component all affect the result. Open MPI then opens the selected Libfabric fabric and domain and creates its endpoints and completion resources.

The common selection parameter is exposed through component-specific aliases. For example, these commands show the provider and selection diagnostics for the matching transport layer and the byte transfer layer:

shell$ mpirun --mca mtl_ofi_verbose 1 \
       --mca mtl_ofi_provider_include cxi ./mpi_hello
shell$ mpirun --mca btl_ofi_verbose 1 \
       --mca btl_ofi_provider_include cxi ./mpi_hello

The exact diagnostics and available component parameters depend on the Open MPI and Libfabric versions in use. ompi_info is the authoritative way to list the parameters in a particular installation.

11.2.1.4. Multiple network interfaces in the ofi BTL

When a node has more network interfaces than local processes, each process of the ofi BTL opens several of them, one BTL module per interface, and the ob1 PML spreads traffic over them. The interfaces are grouped by the PCI root complex they sit under, and each process takes its interfaces from one root complex: driving concurrent traffic through interfaces under more than one root complex can be much slower than using fewer interfaces under one. Contiguous blocks of local ranks share a root complex, which with the usual placement of one rank per GPU puts each rank on the root complex of its GPU.

MCA parameter

Default

Meaning

btl_ofi_modules_per_proc

0

Number of interfaces each process opens; 0 divides them among the local processes. Experimental.

btl_ofi_frag_size

65536

Payload bytes of one fragment, which is also the eager limit and the largest send, clamped to the provider’s maximum message size. Device buffers keep an eager limit of 4 KB unless btl_ofi_accelerator_eager_limit is set.

Messages up to the fragment size are sent eagerly, without the rendezvous round trip. The receiver copies an eager message once more than a rendezvous message, so a stream of messages of 16 KB to 32 KB to one peer can reach more bandwidth with a smaller fragment size, at the cost of latency.

11.2.1.5. LINKx provider

Libfabric also provides a provider named lnx (LINKx). LINKx can combine multiple Libfabric providers or domains behind one tagged-message endpoint. It is an optional Libfabric feature, not an Open MPI-specific network component. Whether it is appropriate depends on the provider versions, the system topology, and the performance goals of the deployment. HPE or the site administrator should confirm that it is recommended for a particular Slingshot installation before it is enabled.

The provider must be configured through the Libfabric environment variable FI_LNX_PROV_LINKS. For example, the following requests one LINKx group containing shared memory and two Slingshot domains:

shell$ export FI_LNX_PROV_LINKS="shm+cxi:cxi0,cxi1"
shell$ mpirun --mca pml cm --mca mtl ofi \
       --mca mtl_ofi_provider_include lnx ./mpi_hello

The provider manual(man fi_lnx) describes additional forms, including multiple groups and provider-specific domain lists. All nodes in the MPI job must use compatible LINKx configurations and the same ordering of linked providers and domains. If shared memory is included, LINKx uses it for intra-node communication; other linked providers are used for off-node communication. LINKx can also distribute messages across multiple linked domains, according to its multi-rail selection policy.

There are important tradeoffs. The LINKx provider described by the current Libfabric manual supports tagged operations, but does not provide hardware offload such as hardware tag matching. Because memory registration does not identify the eventual operation or destination, LINKx registers memory with all linked providers. This can increase memory registration cost and may affect memory behavior. Measure an application with and without LINKx before making it a site-wide default.

11.2.1.6. HPE Slingshot

Open MPI supports HPE Slingshot through the Libfabric cxi provider. The cxi provider maps Open MPI communication operations to the Slingshot Cassini network interface card and Rosetta switch.

Open MPI normally selects its OpenFabrics Interfaces components automatically. To request the tagged-message path explicitly, select the cm PML and ofi MTL, and include the cxi provider. When the selected provider supports hardware tag matching, message matching can be offloaded from the host processor to the network interface card. The cxi provider supports this capability. Tagged messages carry a numeric label that Open MPI uses to match a send operation to the correct receive operation:

shell$ mpirun --mca pml cm --mca mtl ofi \
       --mca mtl_ofi_provider_include cxi ./mpi_hello

Use the Libfabric fi_info utility to confirm that the installed Libfabric can find the cxi provider:

shell$ fi_info -p cxi

Important

When using the HPE CXI provider with mpirun as the job launcher, it is recommended to set the PRRTE ras_base_launch_orted_on_hn MCA parameter to 1 by adding --prtemca ras_base_launch_orted_on_hn 1 to the job launch command line. This ensures that MPI processes launched on the first node of an allocation are able to use the CXI provider.

For more information, refer to the Libfabric web site.

11.2.1.7. Omni-Path: multi-rail with multiple HFI cards

The multi-rail feature lets a single process use multiple Host Fabric Interface (HFI) cards to transfer a message, improving message bandwidth. The PSM2 library provides multi-rail support, which is off by default. The behavior is controlled with the following environment variables:

  • PSM2_MULTIRAIL=[0,1,2]: 0 disables multi-rail, 1 enables it across all HFIs in the system, and 2 enables multi-rail within a NUMA node.

  • PSM2_MULTIRAIL_MAP=unit:port,unit:port,...

These variables may be set on the mpirun command line or in the environment. For example:

shell$ mpirun --mca mtl [psm2|ofi] -x PSM2_MULTIRAIL=1 \
       -n 2 -H host1,host2 ./a.out

Note

When using the OFI MTL, ensure that the PSM2 OFI provider is used for communication with OPA devices.

11.2.1.8. Omni-Path: multi-HFI support in PSM2

Multi-HFI support describes the use of multiple HFIs in a system among the MPI ranks local to a node, in order to load-balance the hardware resources. It differs from the multi-rail feature, which allows a single process to use all HFIs in the system. For an MPI job with multiple processes on a single node, the default PSM2 behavior depends on the affinity settings of the MPI process: the PSM2 library defaults to using the HFI that is in the same NUMA node as the MPI process.

Users can restrict a process to a single HFI with an environment variable:

  • HFI_UNIT=N: valid values of N are 0, 1, 2, and 3.

More details can be found in the PSM2 Programmer’s Guide and the Omni-Path Fabric Performance Tuning Guide; see the Cornelis Networks Customer Center.

11.2.1.9. GPU memory and Libfabric providers

When an MPI buffer resides in graphics processing unit memory, Open MPI can request Libfabric support for heterogeneous memory through the FI_HMEM capability. The ofi matching transport layer and byte transfer layer request this capability by default when it is available in the build. The matching transport layer and byte transfer layer can disable the request with mtl_ofi_disable_hmem and btl_ofi_disable_hmem, respectively.

Requesting heterogeneous-memory support does not guarantee direct network access to every type of GPU memory. The selected provider must advertise the required capability, and the Libfabric build must include support for the GPU runtime in use. If no suitable provider is found with heterogeneous-memory requirements, Open MPI may retry provider discovery without that requirement; this can result in host-memory staging or a provider that does not support direct GPU buffers.

LINKx deserves additional care for GPU buffers. LINKx forwards memory registration to all providers in a link, and the linked providers may have different GPU-memory capabilities. A LINKx configuration should therefore contain only providers that support the intended buffer type and memory registration mode. Validate the configuration with a representative application and consult the installed provider manuals before using GPU buffers in production.

11.2.1.10. Memory monitors

Open MPI and Libfabric both cache memory registrations. Each cache needs a memory monitor that reports when cached memory is unmapped or remapped.

Open MPI’s monitor comes from the memory framework, which has three components:

  • ucm reports the releases that UCX’s memory event layer (UCM) sees. It is only used when UCM has already hooked the C library’s memory functions, for example because a UCX component is loaded.

  • kdreg2 uses the kdreg2 kernel module, which watches each registered region. It is built when configure finds kdreg2.h (see --with-kdreg2).

  • patcher intercepts munmap(2) and related calls.

Open MPI uses the highest-priority component that it can open (see the memory_ucm_priority, memory_kdreg2_priority and memory_patcher_priority MCA parameters), and moves on to the next one when a component cannot be used, such as kdreg2 when its device is missing. Set the memory MCA parameter to choose one; for example, --mca memory kdreg2. If none of the named components can be used, Open MPI prints an error and runs without memory hooks.

The component that Open MPI uses also decides Libfabric’s monitor, as follows:

  • With ucm or patcher, Open MPI exports its monitor to Libfabric (version 1.14 and later), and Libfabric uses it instead of the monitor that FI_MR_CACHE_MONITOR names.

  • With kdreg2, Open MPI does not export a monitor, because kdreg2 only watches the regions that Open MPI registers. Libfabric picks its own monitor, which FI_MR_CACHE_MONITOR controls. When built with kdreg2 support, Libfabric 2.4 and later uses its own kdreg2 monitor by default; Open MPI and Libfabric then each open the kdreg2 device and watch their own regions.

With OMPI_MCA_opal_common_ofi_verbose=1, Open MPI prints a message when it does not export its monitor.