10.4. Examples

Listed here are the subset of command line options that will be used in the process mapping/ranking/binding examples below.

10.4.1. Specifying Host Nodes

Use one of the following options to specify which hosts (nodes) within the PRRTE DVM environment to run on.

--host <host1,host2,...,hostN>

# or

--host <host1:X,host2:Y,...,hostN:Z>

  • List of hosts on which to invoke processes. After each hostname a colon (:) followed by a positive integer can be used to specify the number of slots on that host (:X, :Y, and :Z). The default is 1.

--hostfile <hostfile>

  • Provide a hostfile to use.

10.4.2. Process Mapping / Ranking / Binding Options

  • -c #, -n #, --n #, --np <#>: Run this many copies of the program on the given nodes. This option indicates that the specified file is an executable program and not an application context. If no value is provided for the number of copies to execute (i.e., neither the -np nor its synonyms are provided on the command line), prun will automatically execute a copy of the program on each process slot (see below for description of a “process slot”). This feature, however, can only be used in the SPMD model and will return an error (without beginning execution of the application) otherwise.

    Note

    These options specify the number of processes to launch. None of the options imply a particular binding policy — e.g., requesting N processes for each package does not imply that the processes will be bound to the package.

  • --map-by <object>: Map to the specified object. Supported objects include:

    • slot

    • hwthread

    • core (default)

    • l1cache

    • l2cache

    • l3cache

    • numa

    • package

    • node

    • seq

    • ppr

    • rankfile

    • pe-list

    Any object can include qualifiers by adding a colon (:) and any colon-delimited combination of one or more of the following to the --map-by options:

    • PE=n bind n processing elements to each process (can not be used in combination with rankfile or pe-list directives)

      Error

      JMS Several of the options below refer to pe-list. Is this option supposed to be PE-LIST=n, not PE=n?

    • SPAN load balance the processes across the allocation (cannot be used in combination with slot, node, seq, ppr, rankfile, or pe-list directives)

    • OVERSUBSCRIBE allow more processes on a node than processing elements

    • NOOVERSUBSCRIBE means !OVERSUBSCRIBE

    • NOLOCAL do not launch processes on the same node as prun

    • HWTCPUS use hardware threads as CPU slots

    • CORECPUS use cores as CPU slots (default)

    • INHERIT indicates that a child job (i.e., one spawned from within an application) shall inherit the placement policies of the parent job that spawned it.

    • NOINHERIT means !INHERIT

    • FILE=<path> (path to file containing sequential or rankfile entries).

    • ORDERED only applies to the PE-LIST option to indicate that procs are to be bound to each of the specified CPUs in the order in which they are assigned (i.e., the first proc on a node shall be bound to the first CPU in the list, the second proc shall be bound to the second CPU, etc.)

    ppr policy example: --map-by ppr:N:<object> will launch N times the number of objects of the specified type on each node.

    Note

    Directives and qualifiers are case-insensitive and can be shortened to the minimum number of characters to uniquely identify them. Thus, L1CACHE can be given as l1cache or simply as L1.

  • --rank-by <object>: This assigns ranks in round-robin fashion according to the specified object. The default follows the mapping pattern. Supported rank-by objects include:

    • slot

    • node

    • fill

    • span

    There are no qualifiers for the --rank-by directive.

  • --bindto <object>: This binds processes to the specified object. See defaults in Quick Summary. Supported bindto objects include:

    • none

    • hwthread

    • core

    • l1cache

    • l2cache

    • l3cache

    • numa

    • package

    Any object can include qualifiers by adding a colon (:) and any colon-delimited combination of one or more of the following to the --bindto options:

    • overload-allowed allows for binding more than one process in relation to a CPU

    • if-supported if binding to that object is supported on this system.

10.4.3. Specifying Host Nodes

Host nodes can be identified on the command line with the --host option or in a hostfile.

For example, assuming no other resource manager or scheduler is involved:

prun --host aa,aa,bb ./a.out

This launches two processes on node aa and one on bb.

prun --host aa ./a.out

This launches one process on node aa.

prun --host aa:5 ./a.out

This launches five processes on node aa.

Or, consider the hostfile:

$ cat myhostfile
aa slots=2
bb slots=2
cc slots=2

Here, we list both the host names (aa, bb, and cc) but also how many “slots” there are for each. Slots indicate how many processes can potentially execute on a node. For best performance, the number of slots may be chosen to be the number of cores on the node or the number of processor sockets.

If the hostfile does not provide slots information, the PRRTE DVM will attempt to discover the number of cores (or hwthreads, if the :HWTCPUS qualifier to the --map-by option is set) and set the number of slots to that value.

Examples using the hostfile above with and without the --host option:

prun --hostfile myhostfile ./a.out

This will launch two processes on each of the three nodes.

prun --hostfile myhostfile --host aa ./a.out

This will launch two processes, both on node aa.

prun --hostfile myhostfile --host dd ./a.out

This will find no hosts to run on and abort with an error. That is, the specified host dd is not in the specified hostfile.

When running under resource managers (e.g., SLURM, Torque, etc.), PRTE will obtain both the hostnames and the number of slots directly from the resource manger. The behavior of --host in that environment will behave the same as if a hostfile was provided (since it is provided by the resource manager).

10.4.4. Specifying Number of Processes

As we have just seen, the number of processes to run can be set using the hostfile. Other mechanisms exist.

The number of processes launched can be specified as a multiple of the number of nodes or processor sockets available. Consider the hostfile below for the examples that follow.

$ cat myhostfile
aa
bb

For example:

prun --hostfile myhostfile --map-by ppr:2:package ./a.out

This launches processes 0-3 on node aa and process 4-7 on node bb, where aa and bb are both dual-package nodes. The --map-by ppr:2:package option also turns on the --bindto package option, which is discussed in a later section.

prun --hostfile myhostfile --map-by ppr:2:node ./a.out

This launches processes 0-1 on node aa and processes 2-3 on node bb.

prun --hostfile myhostfile --map-by ppr:1:node ./a.out

This launches one process per host node.

Another alternative is to specify the number of processes with the --np option. Consider now the hostfile:

$ cat myhostfile
aa slots=4
bb slots=4
cc slots=4

With this hostfile:

prun --hostfile myhostfile --np 6 ./a.out

This will launch processes 0-3 on node aa and processes 4-5 on node bb. The remaining slots in the hostfile will not be used since the -np option indicated that only 6 processes should be launched.

10.4.5. Mapping Processes to Devices

Consider a two-socket node with eight NUMA domains and four GPUs, where the GPUs are attached to NUMA domains 1 and 2 on the first socket but 6 and 7 on the second. That asymmetry is the case device mapping exists for: no --mapby numa or --mapby package expression selects those four domains, because they are not at the same position within each socket.

$ prun -n 4 --mapby device=gpu --bindto core ./a.out

places one process per GPU, in PCI bus order, each bound to the first available core in the CPUs local to its own GPU — on the machine described above, cores 16, 32, 96 and 112.

Binding may be to any object at or below the device’s locality:

$ prun -n 1 --mapby device=gpu --bindto numa ./a.out

binds the process to the whole NUMA domain its GPU is attached to, while --bindto l3cache binds it to one L3 cache within that domain and --bindto core to a single core. Asking for --bindto package on this machine is an error: a package contains the GPU’s NUMA domain and three others, so binding there would place the process on CPUs the GPU is not local to.

A device is assigned to a process rather than subdivided between processes, so by default each device takes one process: asking for more processes than there are devices is an error. Where sharing the devices is intended, say so with the shared qualifier:

$ prun -n 8 --mapby device=gpu:shared --bindto core ./a.out

On the four-GPU machine above that runs two processes per GPU. Note the processes still get separate cores: sharing a device and overloading a CPU are different resources and different decisions, which is why they have different qualifiers — shared here, and overload-allowed on --bindto for the CPUs.

Where a job wants its processes spread across sockets rather than filling the first, add the interleave qualifier:

$ prun -n 2 --mapby device=gpu:interleave --bindto core ./a.out

This reorders the device list so that consecutive processes land on different packages — on the machine above, cores 16 and 96 rather than 16 and 32. The level may be given explicitly (interleave=numa, for instance); it defaults to package.

Where a process needs more than one device, ndev says how many:

$ prun -n 2 --mapby device=gpu:ndev=2 --bindto package ./a.out

gives each of the two processes two GPUs. A process holding devices in different NUMA domains is local to neither of them alone, so its locality becomes whatever contains them both — here the package, which is why binding to a package is legal in this case and an error without ndev.

Because the devices are handed out in groups taken in order from the device list, interleave composes with ndev: the interleaving decides the order, and the grouping then takes contiguous runs of it.

The reverse ratio — several processes on each device rather than several devices for each process — is a ppr pattern, spelled the same way as every other:

$ prun --mapby ppr:2:device=gpu --bindto core ./a.out

places two processes on each GPU, each bound to its own core within that GPU’s locality. As with any ppr pattern the process count follows from the pattern when -n is not given: four GPUs yield eight processes.

Naming a single device rather than a class places every process near that one device, which suits a job whose performance depends on one particular fabric interface:

$ prun -n 8 --mapby device=mlx5_0 --bindto core ./a.out

Other classes are selected the same way: device=network for the node’s network interfaces and device=block for block devices. device=nic, device=fabric and device=openfabrics all mean device=network: a card that presents both an OpenFabrics device and a network interface is one device under any of them.

10.4.6. Mapping Processes to Nodes Using Policies

The examples above illustrate the default mapping of process processes to nodes. This mapping can also be controlled with various prun / prterun options that describe mapping policies.

$ cat myhostfile
aa slots=4
bb slots=4
cc slots=4

Consider the hostfile above, with --np 6:

Command

Ranks on aa

Ranks on bb

Ranks on cc

prun

0 1 2 3

4 5

prun --map-by node

0 3

1 4

2 5

prun --map-by node:NOLOCAL

0 2 4

1 3 5

The --map-by node option will load balance the processes across the available nodes, numbering each process by node in a round-robin fashion.

The :NOLOCAL qualifier to --map-by prevents any processes from being mapped onto the local host (in this case node aa). While prun typically consumes few system resources, the :NOLOCAL qualifier can be helpful for launching very large jobs where prun may actually need to use noticeable amounts of memory and/or processing time.

Just as --np can specify fewer processes than there are slots, it can also oversubscribe the slots. For example, with the same hostfile:

prun --hostfile myhostfile --np 14 ./a.out

This will produce an error since the default :NOOVERSUBSCRIBE qualifier to --map-by prevents oversubscription.

To oversubscribe the nodes you can use the :OVERSUBSCRIBE qualifier to --map-by:

prun --hostfile myhostfile --np 14 --map-by :OVERSUBSCRIBE ./a.out

This will launch processes 0-5 on node aa, 6-9 on bb, and 10-13 on cc.

Limits to oversubscription can also be specified in the hostfile itself with the max_slots field:

$ cat myhostfile
aa slots=4 max_slots=4
bb         max_slots=8
cc slots=4

The max_slots field specifies such a limit. When it does, the slots value defaults to the limit. Now:

prun --hostfile myhostfile --np 14 --map-by :OVERSUBSCRIBE ./a.out

This causes the first 12 processes to be launched as before, but the remaining two processes will be forced onto node cc. The other two nodes are protected by the hostfile against oversubscription by this job.

Using the :NOOVERSUBSCRIBE qualifier to --map-by option can be helpful since the PRTE DVM currently does not get max_slots values from the resource manager.

Of course, --np can also be used with the --host option. For example,

prun --host aa,bb --np 8 ./a.out

This will produce an error since the default :NOOVERSUBSCRIBE qualifier to --map-by prevents oversubscription.

prun --host aa,bb --np 8 --map-by :OVERSUBSCRIBE ./a.out

This launches 8 processes. Since only two hosts are specified, after the first two processes are mapped, one to aa and one to bb, the remaining processes oversubscribe the specified hosts evenly.

prun --host aa:2,bb:6 --np 8 ./a.out

This launches 8 processes. Processes 0-1 on node aa since it has 2 slots and processes 2-7 on node bb since it has 6 slots.

And here is a MIMD example:

prun --host aa --np 1 hostname : --host bb,cc --np 2 uptime

This will launch process 0 running hostname on node aa and processes 1 and 2 each running uptime on nodes bb and cc, respectively.