OpenFabrics Interfaces (OFI) / Libfabric support
================================================

"OFI" stands for the `OpenFabrics Interfaces
<https://libfabric.org/>`_, which are implemented in the ``libfabric``
library; the two terms are typically used interchangeably.

Open MPI supports many different underlying networks via Libfabric,
including (but not limited to):

* AWS EFA
* Cisco usNIC
* Cornelis Networks Omni-Path
* HPE Slingshot 11

In general, the OFI-based components in Open MPI auto-select themselves
as appropriate at run time.  The remaining sections describe the
OFI-based components and the tuning options for specific OFI-based
network types.

Libfabric (OFI) components in Open MPI
--------------------------------------

Open MPI has three main components for Libfabric (OFI) communication:

#. The ``ofi`` MTL, available since Open MPI v1.10, is used with the
   ``cm`` PML for two-sided MPI communication (for example,
   ``MPI_Send`` and ``MPI_Recv``).  It requires that the Libfabric
   provider support reliable datagrams with ordered tagged messaging
   |mdash| specifically, ``FI_EP_RDM`` endpoints, ``FI_TAGGED``
   capabilities, and ``FI_ORDER_SAS`` ordering.

#. The ``ofi`` BTL, available since Open MPI v4.0.0, is primarily
   intended for one-sided MPI communication (for example,
   ``MPI_Put``), but can also support BTL send/receive operations.  It
   requires that the Libfabric provider support reliable datagrams, RMA
   and atomic operations, and remote atomic completion notifications
   |mdash| specifically, ``FI_EP_RDM`` endpoints, ``FI_RMA`` and
   ``FI_ATOMIC`` capabilities, and the ``FI_DELIVERY_COMPLETE`` op
   flag.

#. The ``usnic`` BTL is used exclusively with Cisco usNIC-based
   networks.  It auto-selects itself over the other OFI-based
   components when run on such networks.

Consult each Libfabric provider's man page (for example,
``fi_sockets(7)``) to understand which provider will work for each of
the components above.  Some providers must be paired with a Libfabric
utility provider; for example, the verbs provider needs the ``ofi_rxm``
utility provider to supply reliable datagram endpoint support
(``verbs;ofi_rxm``).

Each component has MCA parameters that specify the Libfabric
provider(s) to include in or exclude from the selection process.  For
example:

.. code-block::

   shell$ mpirun --mca pml cm --mca mtl ofi \
          --mca mtl_ofi_provider_include psm2 ./mpi_hello

Each component also has its own component-specific parameters; use
``ompi_info`` to list them.  For example:

.. code-block::

   shell$ ompi_info --param mtl ofi --level 9

Point-to-point selection on EFA nodes with GPUs
-----------------------------------------------

Open MPI initializes every available PML and keeps the one that reports the
highest priority.  The ``cm`` PML reports the priority of its MTL; the ``ofi``
MTL reports 25 by default, and the ``ob1`` PML reports 20.  On most systems
``cm`` with the ``ofi`` MTL is therefore selected wherever Libfabric is
available.

On nodes with GPUs whose MTL would use the ``efa`` provider, the ``ofi`` MTL
reports a priority of 10 instead, so that the ``ob1`` PML with the ``ofi`` BTL
is selected.  The ``ofi`` BTL can drive several network interfaces per process,
where the ``ofi`` MTL drives one, and it can move device buffers with RDMA.
The rule applies when all of the following hold:

* the node has a GPU, as found from the PCI devices in the hwloc topology (a
  display controller from NVIDIA or AMD), which every process on the node
  sees the same way whatever ``CUDA_VISIBLE_DEVICES`` is set to;
* the provider include and exclude settings allow ``efa``, and Libfabric
  offers it;
* the ``mtl_ofi_priority`` MCA parameter has not been set.

Nodes without GPUs and other providers keep the previous selection.  To use
``cm`` with the ``ofi`` MTL on such a node, select it explicitly, or set the
MTL's priority, which disables the rule:

.. code-block:: sh

   shell$ mpirun --mca pml cm ...
   shell$ mpirun --mca mtl_ofi_priority 25 ...

The decision is reported with ``--mca opal_common_ofi_verbose 1``.  All
processes of a job must use the same PML: a job whose nodes differ (for
example, some with GPUs and some without) stops at startup with a PML mismatch
error, and needs ``--mca pml cm`` or ``--mca pml ob1`` on the command line.

Provider and domain selection
-----------------------------

Open MPI asks Libfabric for a list of ``fi_info`` structures that satisfy the
capabilities required by the selected component.  An ``fi_info`` structure
describes a provider, its fabric, and a domain.  In this context, a Libfabric
domain is the provider's view of a usable network resource; on many systems it
corresponds to one network interface card or one provider-specific network
endpoint.  The domain name is reported in the ``fi_info`` structure and is not
necessarily the same as the operating system device name.

The ``ofi`` MTL and the ``ofi`` BTL use the common Open MPI selection code.
The selection process is:

#. Open MPI applies the provider include or exclude setting to the structures
   returned by Libfabric.  These settings select provider names, not arbitrary
   hardware device names.
#. Open MPI selects one compatible provider structure.  When several structures
   describe domains of the same provider with the same requested capabilities,
   those structures are candidates for local-resource selection.
#. If the process is not bound to a set of processing units, Open MPI selects
   among the candidate domains using the process's local rank.  This is a
   round-robin choice intended to balance ranks across domains.
#. If the process is bound and the structures contain usable PCI information,
   Open MPI calculates device locality and selects a domain closest to the
   process.  If several domains are equally close, the process's rank within
   its processor package selects among the tied domains.
#. If accelerator locality is enabled and the accelerator and provider expose
   usable PCI information, Open MPI first attempts to select the domain closest
   to the accelerator.  Ties are distributed using the local rank on that
   accelerator.
#. If locality information is unavailable, Open MPI falls back to the rank-based
   round-robin choice.  A process that is not bound cannot receive a locality
   guarantee from the operating system.

This selection is performed independently by each MPI process.  It does not
mean that every process opens every domain, and it does not guarantee that a
particular operating system interface is selected.  The provider's ``fi_info``
structures, process binding, PCI topology information, and the selected Open MPI
component all affect the result.  Open MPI then opens the selected Libfabric
fabric and domain and creates its endpoints and completion resources.

The common selection parameter is exposed through component-specific aliases.
For example, these commands show the provider and selection diagnostics for
the matching transport layer and the byte transfer layer:

.. code-block:: sh

   shell$ mpirun --mca mtl_ofi_verbose 1 \
          --mca mtl_ofi_provider_include cxi ./mpi_hello
   shell$ mpirun --mca btl_ofi_verbose 1 \
          --mca btl_ofi_provider_include cxi ./mpi_hello

The exact diagnostics and available component parameters depend on the Open
MPI and Libfabric versions in use.  ``ompi_info`` is the authoritative way to
list the parameters in a particular installation.

Multiple network interfaces in the ``ofi`` BTL
----------------------------------------------

When a node has more network interfaces than local processes, each process of
the ``ofi`` BTL opens several of them, one BTL module per interface, and the
``ob1`` PML spreads traffic over them.  The interfaces are grouped by the PCI
root complex they sit under, and each process takes its interfaces from one
root complex: driving concurrent traffic through interfaces under more than one
root complex can be much slower than using fewer interfaces under one.
Contiguous blocks of local ranks share a root complex, which with the usual
placement of one rank per GPU puts each rank on the root complex of its GPU.

.. list-table::
   :header-rows: 1

   * - MCA parameter
     - Default
     - Meaning
   * - ``btl_ofi_modules_per_proc``
     - 0
     - Number of interfaces each process opens; 0 divides them among the
       local processes.  Experimental.
   * - ``btl_ofi_frag_size``
     - 65536
     - Payload bytes of one fragment, which is also the eager limit and the
       largest send, clamped to the provider's maximum message size.  Device
       buffers keep an eager limit of 4 KB unless
       ``btl_ofi_accelerator_eager_limit`` is set.

Messages up to the fragment size are sent eagerly, without the rendezvous
round trip.  The receiver copies an eager message once more than a rendezvous
message, so a stream of messages of 16 KB to 32 KB to one peer can reach more
bandwidth with a smaller fragment size, at the cost of latency.

LINKx provider
--------------

Libfabric also provides a provider named ``lnx`` (LINKx).  LINKx can combine
multiple Libfabric providers or domains behind one tagged-message endpoint. It
is an optional Libfabric feature, not an Open MPI-specific network component.
Whether it is appropriate depends on the provider versions, the system
topology, and the performance goals of the deployment.  HPE or the site
administrator should confirm that it is recommended for a particular
Slingshot installation before it is enabled.

The provider must be configured through the Libfabric environment variable
``FI_LNX_PROV_LINKS``.  For example, the following requests one LINKx group
containing shared memory and two Slingshot domains:

.. code-block:: sh

   shell$ export FI_LNX_PROV_LINKS="shm+cxi:cxi0,cxi1"
   shell$ mpirun --mca pml cm --mca mtl ofi \
          --mca mtl_ofi_provider_include lnx ./mpi_hello

The provider manual(man fi_lnx) describes additional forms, including multiple
groups and provider-specific domain lists.  All nodes in the MPI job must use
compatible LINKx configurations and the same ordering of linked providers and
domains.  If shared memory is included, LINKx uses it for intra-node
communication; other linked providers are used for off-node communication.
LINKx can also distribute messages across multiple linked domains, according
to its multi-rail selection policy.

There are important tradeoffs.  The LINKx provider described by the current
Libfabric manual supports tagged operations, but does not provide hardware
offload such as hardware tag matching.  Because memory registration does not
identify the eventual operation or destination, LINKx registers memory with
all linked providers.  This can increase memory registration cost and may
affect memory behavior.  Measure an application with and without LINKx before
making it a site-wide default.

HPE Slingshot
-------------

Open MPI supports HPE Slingshot through the Libfabric ``cxi`` provider.  The
``cxi`` provider maps Open MPI communication operations to the Slingshot
Cassini network interface card and Rosetta switch.

Open MPI normally selects its OpenFabrics Interfaces components automatically.
To request the tagged-message path explicitly, select the ``cm`` PML and
``ofi`` MTL, and include the ``cxi`` provider.  When the selected provider
supports hardware tag matching, message matching can be offloaded from the
host processor to the network interface card.  The ``cxi`` provider supports
this capability.  Tagged messages carry a numeric label that Open MPI uses to
match a send operation to the correct receive operation:

.. code-block:: sh

   shell$ mpirun --mca pml cm --mca mtl ofi \
          --mca mtl_ofi_provider_include cxi ./mpi_hello

Use the Libfabric ``fi_info`` utility to confirm that the installed Libfabric
can find the ``cxi`` provider:

.. code-block:: sh

   shell$ fi_info -p cxi

.. important:: When using the HPE CXI provider with ``mpirun`` as the
               job launcher, it is recommended to set the PRRTE
               ``ras_base_launch_orted_on_hn`` MCA parameter to 1 by
               adding ``--prtemca ras_base_launch_orted_on_hn 1`` to
               the job launch command line.  This ensures that MPI
               processes launched on the first node of an allocation
               are able to use the CXI provider.

For more information, refer to the `Libfabric web site
<https://libfabric.org/>`_.

Omni-Path: multi-rail with multiple HFI cards
---------------------------------------------

The multi-rail feature lets a single process use multiple Host Fabric
Interface (HFI) cards to transfer a message, improving message
bandwidth.  The PSM2 library provides multi-rail support, which is off
by default.  The behavior is controlled with the following environment
variables:

* ``PSM2_MULTIRAIL=[0,1,2]``: 0 disables multi-rail, 1 enables it
  across all HFIs in the system, and 2 enables multi-rail within a NUMA
  node.
* ``PSM2_MULTIRAIL_MAP=unit:port,unit:port,...``

These variables may be set on the ``mpirun`` command line or in the
environment.  For example:

.. code-block::

   shell$ mpirun --mca mtl [psm2|ofi] -x PSM2_MULTIRAIL=1 \
          -n 2 -H host1,host2 ./a.out

.. note:: When using the OFI MTL, ensure that the PSM2 OFI provider is
          used for communication with OPA devices.

Omni-Path: multi-HFI support in PSM2
------------------------------------

Multi-HFI support describes the use of multiple HFIs in a system among
the MPI ranks local to a node, in order to load-balance the hardware
resources.  It differs from the multi-rail feature, which allows a
single process to use all HFIs in the system.  For an MPI job with
multiple processes on a single node, the default PSM2 behavior depends
on the affinity settings of the MPI process: the PSM2 library defaults
to using the HFI that is in the same NUMA node as the MPI process.

Users can restrict a process to a single HFI with an environment
variable:

* ``HFI_UNIT=N``: valid values of ``N`` are 0, 1, 2, and 3.

More details can be found in the PSM2 Programmer's Guide and the
Omni-Path Fabric Performance Tuning Guide; see the `Cornelis Networks
Customer Center <https://customercenter.cornelisnetworks.com/>`_.

GPU memory and Libfabric providers
----------------------------------

When an MPI buffer resides in graphics processing unit memory, Open MPI can
request Libfabric support for heterogeneous memory through the ``FI_HMEM``
capability.  The ``ofi`` matching transport layer and byte transfer layer
request this capability by default when it is available in the build.  The
matching transport layer and byte transfer layer can disable the request
with ``mtl_ofi_disable_hmem`` and ``btl_ofi_disable_hmem``, respectively.

Requesting heterogeneous-memory support does not guarantee direct network
access to every type of GPU memory.  The selected provider must advertise the
required capability, and the Libfabric build must include support for the GPU
runtime in use.  If no suitable provider is found with heterogeneous-memory
requirements, Open MPI may retry provider discovery without that requirement;
this can result in host-memory staging or a provider that does not support
direct GPU buffers.

LINKx deserves additional care for GPU buffers.  LINKx forwards memory
registration to all providers in a link, and the linked providers may have
different GPU-memory capabilities.  A LINKx configuration should therefore
contain only providers that support the intended buffer type and memory
registration mode.  Validate the configuration with a representative
application and consult the installed provider manuals before using GPU
buffers in production.

Memory monitors
---------------

Open MPI and Libfabric both cache memory registrations.  Each cache
needs a memory monitor that reports when cached memory is unmapped or
remapped.

Open MPI's monitor comes from the ``memory`` framework, which has three
components:

* ``ucm`` reports the releases that UCX's memory event layer (UCM)
  sees.  It is only used when UCM has already hooked the C library's
  memory functions, for example because a UCX component is loaded.
* ``kdreg2`` uses the `kdreg2
  <https://github.com/HewlettPackard/shs-kdreg2>`_ kernel module,
  which watches each registered region.  It is built when
  ``configure`` finds ``kdreg2.h`` (see ``--with-kdreg2``).
* ``patcher`` intercepts ``munmap(2)`` and related calls.

Open MPI uses the highest-priority component that it can open (see
the ``memory_ucm_priority``, ``memory_kdreg2_priority`` and
``memory_patcher_priority`` MCA parameters), and moves on to the next
one when a component cannot be used, such as ``kdreg2`` when its
device is missing.  Set the ``memory`` MCA parameter to choose one;
for example, ``--mca memory kdreg2``.  If none of the named components
can be used, Open MPI prints an error and runs without memory hooks.

The component that Open MPI uses also decides Libfabric's monitor, as
follows:

* With ``ucm`` or ``patcher``, Open MPI exports its monitor to
  Libfabric (version 1.14 and later), and Libfabric uses it instead of
  the monitor that ``FI_MR_CACHE_MONITOR`` names.
* With ``kdreg2``, Open MPI does not export a monitor, because
  ``kdreg2`` only watches the regions that Open MPI registers.
  Libfabric picks its own monitor, which ``FI_MR_CACHE_MONITOR``
  controls.  When built with kdreg2 support, Libfabric 2.4 and later
  uses its own ``kdreg2`` monitor by default; Open MPI and Libfabric
  then each open the kdreg2 device and watch their own regions.

With ``OMPI_MCA_opal_common_ofi_verbose=1``, Open MPI prints a message
when it does not export its monitor.
