11.3.1. Selecting an Accelerator Device before calling MPI_Init

A common problem when using accelerators arises when selecting which GPU should be used by an MPI process. The decision is often based by the rank of that process in MPI_COMM_WORLD. The rank of a process can however only be retrieved after the MPI library has correctly initialized. On the other hand, the accelerator resources initialized during MPI_Init can have some associations with the current device, which will be the default device used by a particular eco-system if not set to a different value.

To circumvent this circular problem, applications are encouraged to make use of the environment variable OMPI_COMM_WORLD_LOCAL_RANK that is set by Open MPI at launch time and can be retrieved before MPI_Init. An example code sample using the HIP programming model looks as follows:

int num_devices;
hipGetDeviceCount(&num_devices);
assert (num_devices > 0);

char* ompi_local_rank = getenv("OMPI_COMM_WORLD_LOCAL_RANK");
if (nullptr != ompi_local_rank) {
     hipSetDevice(atoi(ompi_local_rank) % num_devices);
}

MPI_Init (&argc, &argv);
...

Note

Open MPI currently assumes that an MPI processes is using a single accelerator device. Certain software stacks might be able to support multiple GPUs per rank.