# -*- text -*-
#
# Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
#                         University Research and Technology
#                         Corporation.  All rights reserved.
# Copyright (c) 2004-2005 The University of Tennessee and The University
#                         of Tennessee Research Foundation.  All rights
#                         reserved.
# Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
#                         University of Stuttgart.  All rights reserved.
# Copyright (c) 2004-2005 The Regents of the University of California.
#                         All rights reserved.
# Copyright (c) 2011-2020 Cisco Systems, Inc.  All rights reserved
# Copyright (c) 2011      Los Alamos National Security, LLC.
#                         All rights reserved.
# Copyright (c) 2014-2020 Intel, Inc.  All rights reserved.
# Copyright (c) 2021-2026 Nanook Consulting  All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
# This is the US/English general help file for PRRTE's prun.
#
[placement]


Overview
--------

NOTE: PRRTE accepts both the new "--mapby", "--rankby", and "--bindto"
cmd line options, and the older deprecated "--map-by", "--rank-by", and
"--bind-to" versions. For simplicity, the following description will
refer to the new forms.


PRRTE provides a set of three controls for assigning process locations
and ranks:

1. Mapping: Assigns a default location to each process

2. Ranking: Assigns a unique integer rank value to each process

3. Binding: Constrains each process to run on specific processors

These three controls can be specified at the job level (applying to
all application contexts in the job) or, in a multi-program multiple-
data (MPMD) job, independently for each application context separated
by ":" on the "prun" command line. See the "Per-App-Context Mapping"
section under "fundamentals" for details.

This section provides an overview of these three controls.  Unless
otherwise this behavior is shared by "prun(1)" (working with a PRRTE
DVM), and "prterun(1)". More detail about PRRTE process placement is
available in the following sections (using "--help
placement-<section>"):

* "examples": some examples of the interactions between mapping,
  ranking, and binding options.

* "fundamentals": provides deeper insight into PRRTE's mapping,
  ranking, and binding options.

* "limits": explains the difference between *overloading* and
  *oversubscribing* resources.

* "diagnostics": describes options for obtaining various diagnostic
  reports that aid the user in verifying and tuning the placement for
  a specific job.

* "rankfiles": explains the format and use of the rankfile mapper for
  specifying arbitrary process placements.

* "deprecated": a list of deprecated options and their new
  equivalents.

* "all": outputs all the placement help except for the "deprecated"
  and "rankfile" sections.


Quick Summary
=============

The two binaries that most influence process layout are "prte(1)" and
"prun(1)".  The "prte(1)" process discovers the allocation,
establishes a Distributed Virtual Machine by starting a "prted(1)"
daemon on each node of the allocation, and defines the efault
mapping/ranking/binding policies for all jobs.  The "prun(1)" process
defines the specific mapping/ranking/binding for a specific job. Most
of the command line controls are targeted to "prun(1)" since each job
has its own unique requirements.

"prterun(1)" is just a wrapper around "prte(1)" for a single job PRRTE
DVM. It is doing the job of both "prte(1)" and "prun(1)", and, as
such, accepts the sum all of their command line arguments. Any example
that uses "prun(1)" can substitute the use of "prterun(1)" except
where otherwise noted.

The "prte(1)" process attempts to automatically discover the nodes in
the allocation by querying supported resource managers. If a supported
resource manager is not present then "prte(1)" relies on a hostfile
provided by the user.  In the absence of such a hostfile it will run
all processes on the localhost.

If running under a supported resource manager, the "prte(1)" process
will start the daemon processes ("prted(1)") on the remote nodes using
the corresponding resource manager process starter. If no such starter
is available then "ssh" (or "rsh") is used.

Minus user direction, PRRTE will automatically map processes in a
round-robin fashion by CPU, binding each process to its own CPU. The
type of CPU used (core vs hwthread) is determined by (in priority
order):

* user directive on the command line via the HWTCPUS qualifier to the
  "--mapby" directive

* setting the "rmaps_default_mapping_policy" MCA parameter to include
  the "HWTCPUS" qualifier. This parameter sets the default value for a
  PRRTE DVM — qualifiers are carried across to DVM jobs started via
  "prun" unless overridden by the user's command line

* defaulting to "CORE" in topologies where core CPUs are defined, and
  to "hwthreads" otherwise.

By default, the ranks are assigned in accordance with the mapping
directive — e.g., jobs that are mapped by-node will have the process
ranks assigned round-robin on a per-node basis.

PRRTE automatically binds processes unless directed not to do so by
the user. Minus direction, PRRTE will bind individual processes to
their own CPU within the object to which they were mapped. Should a
node become oversubscribed during the mapping process, and if
oversubscription is allowed, all subsequent processes assigned to that
node will *not* be bound.


Definition of 'slot'
--------------------

The term "slot" is used extensively in the rest of this documentation.
A slot is an allocation unit for a process.  The number of slots on a
node indicate how many processes can potentially execute on that node.
By default, PRRTE will allow one process per slot.

If PRRTE is not explicitly told how many slots are available on a node
(e.g., if a hostfile is used and the number of slots is not specified
for a given node), it will determine a maximum number of slots for
that node in one of two ways:

1. Default behavior: By default, PRRTE will attempt to discover the
   number of processor cores on the node, and use that as the number
   of slots available.

2. When "--use-hwthread-cpus" is used: If "--use-hwthread-cpus" is
   specified on the command line, then PRRTE will attempt to discover
   the number of hardware threads on the node, and use that as the
   number of slots available.

This default behavior also occurs when specifying the "--host" option
with a single host.  Thus, the command:

   shell$ prun --host node1 ./a.out

launches a number of processes equal to the number of cores on node
"node1", whereas:

   shell$ prun --host node1 --use-hwthread-cpus ./a.out

launches a number of processes equal to the number of hardware threads
on "node1".

When PRRTE applications are invoked in an environment managed by a
resource manager (e.g., inside of a Slurm job), and PRRTE was built
with appropriate support for that resource manager, then PRRTE will be
informed of the number of slots for each node by the resource manager.
For example:

   shell$ prun ./a.out

launches one process for every slot (on every node) as dictated by the
resource manager job specification.

Also note that the one-process-per-slot restriction can be overridden
in unmanaged environments (e.g., when using hostfiles without a
resource manager) if oversubscription is enabled (by default, it is
disabled).  Most parallel applications and HPC environments do not
oversubscribe; for simplicity, the majority of this documentation
assumes that oversubscription is not enabled.


Slots are not hardware resources
================================

Slots are frequently incorrectly conflated with hardware resources. It
is important to realize that slots are an entirely different metric
than the number (and type) of hardware resources available.

Here are some examples that may help illustrate the difference:

1. More processor cores than slots: Consider a resource manager job
   environment that tells PRRTE that there is a single node with 20
   processor cores and 2 slots available.  By default, PRRTE will only
   let you run up to 2 processes.

   Meaning: you run out of slots long before you run out of processor
   cores.

2. More slots than processor cores: Consider a hostfile with a single
   node listed with a "slots=50" qualification.  The node has 20
   processor cores.  By default, PRRTE will let you run up to 50
   processes.

   Meaning: you can run many more processes than you have processor
   cores.


Definition of "processor element"
---------------------------------

By default, PRRTE defines that a "processing element" is a processor
core.  However, if "--use-hwthread-cpus" is specified on the command
line, then a "processing element" is a hardware thread.
#
[placement-all]

#include#help-placement.txt#placement-fundamentals

#include#help-placement.txt#placement-limits

#include#help-placement.txt#placement-examples

#include#help-placement.txt#placement-diagnostics

#
[placement-examples]


Examples
--------

Listed here are the subset of command line options that will be used
in the process mapping/ranking/binding examples below.


Specifying Host Nodes
=====================

Use one of the following options to specify which hosts (nodes) within
the PRRTE DVM environment to run on.

   --host <host1,host2,...,hostN>

   # or

   --host <host1:X,host2:Y,...,hostN:Z>

* List of hosts on which to invoke processes. After each hostname a
  colon (":") followed by a positive integer can be used to specify
  the number of slots on that host (":X", ":Y", and ":Z"). The default
  is "1".

   --hostfile <hostfile>

* Provide a hostfile to use.


Process Mapping / Ranking / Binding Options
===========================================

* "-c #", "-n #", "--n #", "--np <#>": Run this many copies of the
  program on the given nodes. This option indicates that the specified
  file is an executable program and not an application context. If no
  value is provided for the number of copies to execute (i.e., neither
  the "-np" nor its synonyms are provided on the command line), "prun"
  will automatically execute a copy of the program on each process
  slot (see below for description of a "process slot"). This feature,
  however, can only be used in the SPMD model and will return an error
  (without beginning execution of the application) otherwise.

  Note:

    These options specify the number of processes to launch. None of
    the options imply a particular binding policy — e.g., requesting
    "N" processes for each package does not imply that the processes
    will be bound to the package.

* "--mapby <object>": Map to the specified object. Supported objects
  include:

  * "slot"

  * "hwthread"

  * "core" (default)

  * "l1cache"

  * "l2cache"

  * "l3cache"

  * "numa"

  * "package"

  * "node"

  * "seq"

  * "ppr"

  * "rankfile"

  * "pe-list"

  * "device=<class|name>"

  Any object can include qualifiers by adding a colon (":") and any
  colon-delimited combination of one or more of the following to the "
  --mapby" options:

  * "PE=n" bind "n" processing elements to each process (can not be
    used in combination with rankfile or pe-list directives)

    Error:

      JMS Several of the options below refer to "pe-list". Is this
      option supposed to be "PE-LIST=n", not "PE=n"?

  * "SPAN" load balance the processes across the allocation (cannot be
    used in combination with "slot", "node", "seq", "ppr", "rankfile",
    or "pe-list" directives)

  * "OVERSUBSCRIBE" allow more processes on a node than processing
    elements

  * "NOOVERSUBSCRIBE" means "!OVERSUBSCRIBE"

  * "NOLOCAL" do not launch processes on the same node as "prun"

  * "HWTCPUS" use hardware threads as CPU slots

  * "CORECPUS" use cores as CPU slots (default)

  * "INHERIT" indicates that a child job (i.e., one spawned from
    within an application) shall inherit the placement policies of the
    parent job that spawned it.

  * "NOINHERIT" means "!INHERIT"

  * "FILE=<path>" (path to file containing sequential or rankfile
    entries).

  * "INTERLEAVE[=<level>]" only applies to the "device" directive;
    reorders the device list so consecutive procs land on different
    objects of the given level (default "package")

  * "SHARED[=true|false]" only applies to the "device" directive;
    permits several procs to be assigned the same device (default
    false)

  * "NDEV=<n>" only applies to the "device" directive; assigns n
    devices to each proc rather than one. The proc is then local to
    whatever contains all of them - two GPUs on different NUMA domains
    make their package the locality - so a binding refused with one
    device per proc may be legitimate with several

  * "ORDERED" only applies to the PE-LIST option to indicate that
    procs are to be bound to each of the specified CPUs in the order
    in which they are assigned (i.e., the first proc on a node shall
    be bound to the first CPU in the list, the second proc shall be
    bound to the second CPU, etc.)

  "ppr" policy example: "--mapby ppr:N:<object>" will launch "N"
  times the number of objects of the specified type on each node.

  Note:

    Directives and qualifiers are case-insensitive and can be
    shortened to the minimum number of characters to uniquely identify
    them. Thus, "L1CACHE" can be given as "l1cache" or simply as "L1".

* "--rank-by <object>": This assigns ranks in round-robin fashion
  according to the specified object. The default follows the mapping
  pattern. Supported rank-by objects include:

  * "slot"

  * "node"

  * "fill"

  * "span"

  There are no qualifiers for the "--rank-by" directive.

* "--bindto <object>": This binds processes to the specified object.
  See defaults in Quick Summary.  Supported bindto objects include:

  * "none"

  * "hwthread"

  * "core"

  * "l1cache"

  * "l2cache"

  * "l3cache"

  * "numa"

  * "package"

  Any object can include qualifiers by adding a colon (":") and any
  colon-delimited combination of one or more of the following to the "
  --bindto" options:

  * "overload-allowed" allows for binding more than one process in
    relation to a CPU

  * "if-supported" if binding to that object is supported on this
    system.


Specifying Host Nodes
=====================

Host nodes can be identified on the command line with the "--host"
option or in a hostfile.

For example, assuming no other resource manager or scheduler is
involved:

   prun --host aa,aa,bb ./a.out

This launches two processes on node "aa" and one on "bb".

   prun --host aa ./a.out

This launches one process on node "aa".

   prun --host aa:5 ./a.out

This launches five processes on node "aa".

Or, consider the hostfile:

   $ cat myhostfile
   aa slots=2
   bb slots=2
   cc slots=2

Here, we list both the host names ("aa", "bb", and "cc") but also how
many "slots" there are for each. Slots indicate how many processes can
potentially execute on a node. For best performance, the number of
slots may be chosen to be the number of cores on the node or the
number of processor sockets.

If the hostfile does not provide slots information, the PRRTE DVM will
attempt to discover the number of cores (or hwthreads, if the
":HWTCPUS" qualifier to the "--mapby" option is set) and set the
number of slots to that value.

Examples using the hostfile above with and without the "--host"
option:

   prun --hostfile myhostfile ./a.out

This will launch two processes on each of the three nodes.

   prun --hostfile myhostfile --host aa ./a.out

This will launch two processes, both on node "aa".

   prun --hostfile myhostfile --host dd ./a.out

This will find no hosts to run on and abort with an error. That is,
the specified host "dd" is not in the specified hostfile.

When running under resource managers (e.g., SLURM, Torque, etc.), PRTE
will obtain both the hostnames and the number of slots directly from
the resource manger. The behavior of "--host" in that environment will
behave the same as if a hostfile was provided (since it is provided by
the resource manager).


Specifying Number of Processes
==============================

As we have just seen, the number of processes to run can be set using
the hostfile. Other mechanisms exist.

The number of processes launched can be specified as a multiple of the
number of nodes or processor sockets available. Consider the hostfile
below for the examples that follow.

   $ cat myhostfile
   aa
   bb

For example:

   prun --hostfile myhostfile --mapby ppr:2:package ./a.out

This launches processes 0-3 on node "aa" and process 4-7 on node "bb",
where "aa" and "bb" are both dual-package nodes. The "--mapby
ppr:2:package" option also turns on the "--bindto package" option,
which is discussed in a later section.

   prun --hostfile myhostfile --mapby ppr:2:node ./a.out

This launches processes 0-1 on node "aa" and processes 2-3 on node
"bb".

   prun --hostfile myhostfile --mapby ppr:1:node ./a.out

This launches one process per host node.

Another alternative is to specify the number of processes with the "--
np" option. Consider now the hostfile:

   $ cat myhostfile
   aa slots=4
   bb slots=4
   cc slots=4

With this hostfile:

   prun --hostfile myhostfile --np 6 ./a.out

This will launch processes 0-3 on node "aa" and processes 4-5 on node
"bb".  The remaining slots in the hostfile will not be used since the
"-np" option indicated that only 6 processes should be launched.


Mapping Processes to Nodes Using Policies
=========================================

The examples above illustrate the default mapping of process processes
to nodes. This mapping can also be controlled with various "prun" /
"prterun" options that describe mapping policies.

   $ cat myhostfile
   aa slots=4
   bb slots=4
   cc slots=4

Consider the hostfile above, with "--np 6":

+---------------------------+---------------------------+---------------------------+---------------------------+
| Command                   | Ranks on "aa"             | Ranks on "bb"             | Ranks on "cc"             |
|===========================|===========================|===========================|===========================|
| "prun"                    | 0 1 2 3                   | 4 5                       |                           |
+---------------------------+---------------------------+---------------------------+---------------------------+
| "prun --mapby node"       | 0 3                       | 1 4                       | 2 5                       |
+---------------------------+---------------------------+---------------------------+---------------------------+
| "prun --mapby             |                           | 0 2 4                     | 1 3 5                     |
| node:NOLOCAL"             |                           |                           |                           |
+---------------------------+---------------------------+---------------------------+---------------------------+

The "--mapby node" option will load balance the processes across the
available nodes, numbering each process by node in a round-robin
fashion.

The ":NOLOCAL" qualifier to "--mapby" prevents any processes from
being mapped onto the local host (in this case node "aa"). While
"prun" typically consumes few system resources, the ":NOLOCAL"
qualifier can be helpful for launching very large jobs where "prun"
may actually need to use noticeable amounts of memory and/or
processing time.

Just as "--np" can specify fewer processes than there are slots, it
can also oversubscribe the slots. For example, with the same hostfile:

   prun --hostfile myhostfile --np 14 ./a.out

This will produce an error since the default ":NOOVERSUBSCRIBE"
qualifier to "--mapby" prevents oversubscription.

To oversubscribe the nodes you can use the ":OVERSUBSCRIBE" qualifier
to "--mapby":

   prun --hostfile myhostfile --np 14 --mapby :OVERSUBSCRIBE ./a.out

This will launch processes 0-5 on node "aa", 6-9 on "bb", and 10-13 on
"cc".

Limits to oversubscription can also be specified in the hostfile
itself with the "max_slots" field:

   $ cat myhostfile
   aa slots=4 max_slots=4
   bb         max_slots=8
   cc slots=4

The "max_slots" field specifies such a limit. When it does, the
"slots" value defaults to the limit. Now:

   prun --hostfile myhostfile --np 14 --mapby :OVERSUBSCRIBE ./a.out

This causes the first 12 processes to be launched as before, but the
remaining two processes will be forced onto node cc. The other two
nodes are protected by the hostfile against oversubscription by this
job.

Using the ":NOOVERSUBSCRIBE" qualifier to "--mapby" option can be
helpful since the PRTE DVM currently does not get "max_slots" values
from the resource manager.

Of course, "--np" can also be used with the "--host" option. For
example,

   prun --host aa,bb --np 8 ./a.out

This will produce an error since the default ":NOOVERSUBSCRIBE"
qualifier to "--mapby" prevents oversubscription.

   prun --host aa,bb --np 8 --mapby :OVERSUBSCRIBE ./a.out

This launches 8 processes. Since only two hosts are specified, after
the first two processes are mapped, one to "aa" and one to "bb", the
remaining processes oversubscribe the specified hosts evenly.

   prun --host aa:2,bb:6 --np 8 ./a.out

This launches 8 processes. Processes 0-1 on node "aa" since it has 2
slots and processes 2-7 on node "bb" since it has 6 slots.

And here is a MIMD example:

   prun --host aa --np 1 hostname : --host bb,cc --np 2 uptime

This will launch process 0 running "hostname" on node "aa" and
processes 1 and 2 each running "uptime" on nodes "bb" and "cc",
respectively.


Mapping to Devices
==================

Consider a two-socket node with eight NUMA domains and four GPUs,
where the GPUs are attached to NUMA domains 1 and 2 on the first
socket but 6 and 7 on the second. This asymmetry is the case device
mapping exists for: no "--mapby numa" or "--mapby package" expression
selects those four domains, because they are not at the same position
within each socket.

   $ prun -n 4 --mapby device=gpu --bindto core ./a.out

places one process per GPU, in PCI bus order, each bound to the first
available core in the CPUs local to its own GPU - on the machine
described above, cores 16, 32, 96, and 112.

Binding may be any object at or below the device's locality:

   $ prun -n 1 --mapby device=gpu --bindto numa ./a.out

binds the process to the whole NUMA domain its GPU is attached to,
while "--bindto l3cache" binds it to one L3 cache within that domain,
and "--bindto core" to a single core. Asking for "--bindto package"
on this machine is an error: a package contains the GPU's NUMA domain
and three others, so binding there would place the process on CPUs the
GPU is not local to.

A device is assigned to a process rather than subdivided between
processes, so by default each device takes one process: asking for more
processes than there are devices is an error. Where sharing the devices
is intended, say so with the "shared" qualifier:

   $ prun -n 8 --mapby device=gpu:shared --bindto core ./a.out

On the four-GPU machine above that runs two processes per GPU. Note the
processes still get separate cores: sharing a device and overloading a
CPU are different resources and different decisions, which is why they
have different qualifiers - "shared" here, and "overload-allowed" on
"--bindto" for the CPUs.

Where a job wants its processes spread across sockets rather than
filling the first, add the "interleave" qualifier:

   $ prun -n 2 --mapby device=gpu:interleave --bindto core ./a.out

This reorders the device list so consecutive processes land on
different packages - on the machine above, cores 16 and 96 rather than
16 and 32. The level may be given explicitly ("interleave=numa", for
instance); it defaults to "package".

Where a process needs more than one device, "ndev" says how many:

   $ prun -n 2 --mapby device=gpu:ndev=2 --bindto package ./a.out

gives each of the two processes two GPUs. A process holding devices in
different NUMA domains is local to neither of them alone, so its
locality becomes whatever contains them both - here the package, which
is why binding to a package is legal in this case and an error without
"ndev".

Because the devices are handed out in groups taken in order from the
device list, "interleave" composes with "ndev": the interleaving decides
the order, and the grouping then takes contiguous runs of it.

Mapping by a GPU requires one thing of the machine that the other
device classes do not: that the GPUs can be named to the vendor's
runtime. hwloc learns a GPU's vendor identity - an NVIDIA "GPU-<uuid>"
and its AMD and Intel equivalents - only from that vendor's backend
(NVML, RSMI, Level Zero), which has to be enabled when hwloc is built.
Without it, the GPUs are still discovered and processes are still
placed correctly beside them, but no process can be told which GPU it
was given in terms the library it links will accept. PRRTE refuses the
request in that case. Mapping and then quietly telling the process
nothing looks identical to a working job - the map is right, and the
only symptom is that every process on the node uses the same GPU.

The hwloc that decides this is the one PRRTE was built against, since
each daemon discovers its own node; installing another hwloc alongside
does not change it. Network, fabric and block devices are unaffected,
being named by identifiers hwloc always has.

Where the identity is available, each process is also handed its GPUs in
the environment variable its vendor's runtime reads:

   CUDA_VISIBLE_DEVICES   NVIDIA
   ROCR_VISIBLE_DEVICES   AMD
   ZE_AFFINITY_MASK       Intel

named, for NVIDIA and AMD, by the vendor's own identifier rather than by
an index; Intel is the exception, and is described below. Only processes
actually mapped against a device are given one, and a value already in
the environment is replaced - "--map-by device=" is the more specific
request, and because the identifiers name devices absolutely they compose
correctly with a set a resource manager has already narrowed.

PRRTE never sets the vendor's device ORDERING variable
(CUDA_DEVICE_ORDER and its equivalents). The identifiers do not depend
on it, which is precisely why they are used, and changing it would
renumber every device for the rest of the process's life.

Intel is the exception to "identity rather than index". ZE_AFFINITY_MASK
has no identifier form - it takes Level Zero device ordinals - but the
ordinals are not guessed. hwloc's Level Zero backend records the driver
and device index that zeDeviceGet returned for each device, so the value
is read from that enumeration rather than predicted, and it is read on
the node that will run the process.

An ordinal is not a complete statement on its own: a Level Zero driver
reads ZE_FLAT_DEVICE_HIERARCHY first and then interprets the mask against
the devices that model exposes, so the same ordinals name a card under
COMPOSITE and a tile under FLAT. The model is therefore stated alongside
the mask whenever the process's environment does not already name one.
If it does name one and it disagrees, nothing is set and a message says
so - overriding a deliberate choice would change how many devices the
program sees, and writing a mask that will be read under a different
model would silently hand it half the hardware it was assigned.

A process mapped against a network device is handed it the same way, in
the environment variables the fabric libraries read:

   NCCL_IB_HCA          Mellanox / NVIDIA InfiniBand adapters
   UCX_NET_DEVICES      Mellanox / NVIDIA InfiniBand adapters
   PSM3_NIC             Intel Omni-Path adapters

named as those libraries name the device - "mlx5_0" - which is the name
hwloc gave it. Unlike a GPU's vendor identity that name is always there,
so the identity check above does not apply to this class.

No variable that names an adapter by its unit NUMBER is set - HFI_UNIT
and FI_OPX_HFI_SELECT among them. A unit number is meaningful only
against the enumeration it came from, which is the driver's rather than
one PRRTE performed, and a wrong number in these variables does not fail
- it quietly puts the process on a different adapter. Nothing is set at
all for an adapter whose fabric has no support behind it, rather than
guessing at a variable it might read.

The assignment can also be read directly, whether or not a device
variable was set, from the process's own job data under PMIX_DEVICE_ID.

The reverse ratio - several processes on each device rather than several
devices for each process - is a "ppr" pattern, spelled the same way as
every other:

   $ prun --mapby ppr:2:device=gpu --bindto core ./a.out

places two processes on each GPU, each bound to its own core within that
GPU's locality. As with any ppr pattern, the process count follows from
the pattern when "-n" is not given: four GPUs yield eight processes.

Naming a single device instead of a class places every process near
that one device, which is useful for a job whose performance depends
on one particular fabric interface:

   $ prun -n 8 --mapby device=mlx5_0 --bindto core ./a.out

Other classes are selected the same way - "device=network" for the
node's network interfaces and "device=block" for block devices.
"device=nic", "device=fabric" and "device=openfabrics" all mean
"device=network": a card that presents both an OpenFabrics device
and a network interface is one device under any of them.


Per-App-Context Mapping Example
================================

In an MPMD job, each application context separated by ":" may carry
its own "--mapby", "--rank-by", and "--bindto" directives. Using the
same hostfile:

   $ cat myhostfile
   aa slots=4
   bb slots=4
   cc slots=4

   prun --hostfile myhostfile \
       -n 6 app1 --mapby core --bindto core \
       : \
       -n 2 app2 --mapby node --rank-by fill --bindto none

This will:

* Map "app1"'s 6 processes by core, binding each to its own core
  (processes 0-5).

* Map "app2"'s 2 processes by node in round-robin fashion, leaving
  them unbound (processes 6-7).

Note that process ranks are globally contiguous across both apps: "app1"
receives ranks 0-5 and "app2" receives ranks 6-7. That holds for the
mappers that number their own processes too: a "rankfile" or "seq" file
given to an app numbers THAT app's ranks, and PRRTE offsets them into
the job's numbering.

A directive is per-app only when it is written on an app other than the
first. Written on the first app and nowhere else it describes the whole
job, however many apps follow:

   prun --hostfile myhostfile --mapby core -n 6 app1 : -n 2 app2

maps both apps by core, while

   prun --hostfile myhostfile -n 6 app1 : --mapby core -n 2 app2

maps only "app2" by core and leaves "app1" to the default rules.

The ":NOLOCAL" qualifier may also be applied per app context:

   prun --hostfile myhostfile \
       -n 6 app1 --mapby slot:nolocal \
       : \
       -n 1 app2 --mapby slot

Here "app1" avoids the head node (whichever node "prun" is running on)
while "app2" may run on any node including the head node.
#
[placement-rankfiles]


Rankfiles
---------

Another way to specify arbitrary mappings is with a rankfile, which
gives you detailed control over process binding as well.

Rankfiles are text files that specify detailed information about how
individual processes should be mapped to nodes, and to which
processor(s) they should be bound. Each line of a rankfile specifies
the location of one process. The general form of each line in the
rankfile is:

   rank <N>=<hostname> slot=<slot list>

For example:

   $ cat myrankfile
   rank 0=aa slot=10-12
   rank 1=bb slot=0,1,4
   rank 2=cc slot=1-2
   $ prun --host aa,bb,cc,dd --mapby rankfile:FILE=myrankfile ./a.out

Means that:

* Rank 0 runs on node aa, bound to logical cores 10-12.

* Rank 1 runs on node bb, bound to logical cores 0, 1, and 4.

* Rank 2 runs on node cc, bound to logical cores 1 and 2.

Similarly:

   $ cat myrankfile
   rank 0=aa slot=1:0-2
   rank 1=bb slot=0:0,1,4
   rank 2=cc slot=1-2
   $ prun --host aa,bb,cc,dd --mapby rankfile:FILE=myrankfile ./a.out

Means that:

* Rank 0 runs on node aa, bound to logical package 1, cores 10-12 (the
  0th through 2nd cores on that package).

* Rank 1 runs on node bb, bound to logical package 0, cores 0, 1, and
  4.

* Rank 2 runs on node cc, bound to logical cores 1 and 2.

The hostnames listed above are "absolute," meaning that actual
resolvable hostnames are specified. However, hostnames can also be
specified as "relative," meaning that they are specified in relation
to an externally-specified list of hostnames (e.g., by "prun"'s "--
host" argument, a hostfile, or a job scheduler).

The "relative" specification is of the form ""+n<X>"", where "X" is an
integer specifying the Xth hostname in the set of all available
hostnames, indexed from 0. For example:

   $ cat myrankfile
   rank 0=+n0 slot=10-12
   rank 1=+n1 slot=0,1,4
   rank 2=+n2 slot=1-2
   $ prun --host aa,bb,cc,dd --mapby rankfile:FILE=myrankfile ./a.out

All package/core slot locations are be specified as *logical* indexes.
You can use tools such as HWLOC's "lstopo" to find the logical indexes
of packages and cores.
#
[placement-deprecated]


Deprecated options
------------------

These deprecated options will be removed in a future release.

+----------------------+----------------------+--------------------------------+
| Deprecated Option    | Replacement          | Description                    |
|======================|======================|================================|
| "--bind-to-core"     | "--bindto core"      | Bind processes to cores        |
+----------------------+----------------------+--------------------------------+
| "--bind-to-socket"   | "--bindto package"   | Bind processes to processor    |
|                      |                      | sockets                        |
+----------------------+----------------------+--------------------------------+
| "--bycore"           | "--mapby core"       | Map processes by core          |
+----------------------+----------------------+--------------------------------+
| "--bynode"           | "--mapby node"       | Launch processes one per node, |
|                      |                      | cycling by node in a round-    |
|                      |                      | robin fashion. This spreads    |
|                      |                      | processes evenly among nodes   |
|                      |                      | and assigns ranks in a round-  |
|                      |                      | robin, "by node" manner.       |
+----------------------+----------------------+--------------------------------+
| "--byslot"           | "--mapby slot"       | Map and rank processes round-  |
|                      |                      | robin by slot                  |
+----------------------+----------------------+--------------------------------+
| "--cpus-per-proc     | *--mapby <obj>:PE=   | Bind each process to the       |
| <#perproc>"          | <#perproc>`*         | specified number of CPUs       |
+----------------------+----------------------+--------------------------------+
| "--cpus-per-rank     | "--mapby             | Alias for "--cpus-per-proc"    |
| <#perrank>"          | <obj>:PE=<#perrank>" |                                |
+----------------------+----------------------+--------------------------------+
| "--display-          | "--display ALLOC"    | Display the detected resource  |
| allocation"          |                      | allocation                     |
+----------------------+----------------------+--------------------------------+
| "-display-devel-map" | "--display MAP-      | Display a detailed process map |
|                      | DEVEL"               | (mostly intended for           |
|                      |                      | developers) just before        |
|                      |                      | launch.                        |
+----------------------+----------------------+--------------------------------+
| "--display-map"      | "--display MAP"      | Display a table showing the    |
|                      |                      | mapped location of each        |
|                      |                      | process prior to launch.       |
+----------------------+----------------------+--------------------------------+
| "--display-topo"     | "--display TOPO"     | Display the topology as part   |
|                      |                      | of the process map (mostly     |
|                      |                      | intended for developers) just  |
|                      |                      | before launch.                 |
+----------------------+----------------------+--------------------------------+
| "--do-not-launch"    | "--mapby             | Perform all necessary          |
|                      | :DONOTLAUNCH"        | operations to prepare to       |
|                      |                      | launch the application, but do |
|                      |                      | not actually launch it         |
|                      |                      | (usually used to test mapping  |
|                      |                      | patterns).                     |
+----------------------+----------------------+--------------------------------+
| "--do-not-resolve"   | "--mapby             | Do not attempt to resolve      |
|                      | :DONOTRESOLVE"       | interfaces — usually used to   |
|                      |                      | determine proposed process     |
|                      |                      | placement/binding prior to     |
|                      |                      | obtaining an allocation.       |
+----------------------+----------------------+--------------------------------+
| "-N <num>"           | "--mapby             | Launch "num" processes per     |
|                      | prr:<num>:node"      | node on all allocated nodes    |
+----------------------+----------------------+--------------------------------+
| "--nolocal"          | "--mapby :NOLOCAL"   | Do not run any copies of the   |
|                      |                      | launched application on the    |
|                      |                      | same node as "prun" is         |
|                      |                      | running. This option will      |
|                      |                      | override listing the           |
|                      |                      | "localhost" with "--host" or   |
|                      |                      | any other host-specifying      |
|                      |                      | mechanism.                     |
+----------------------+----------------------+--------------------------------+
| "--nooversubscribe"  | "--mapby             | Do not oversubscribe any       |
|                      | :NOOVERSUBSCRIBE"    | nodes; error (without starting |
|                      |                      | any processes) if the          |
|                      |                      | requested number of processes  |
|                      |                      | would cause oversubscription.  |
|                      |                      | This option implicitly sets    |
|                      |                      | "max_slots" equal to the       |
|                      |                      | "slots" value for each node.   |
|                      |                      | (Enabled by default).          |
+----------------------+----------------------+--------------------------------+
| "--npernode          | "--mapby             | On each node, launch this many |
| <#pernode>"          | ppr:<#pernode>:node" | processes                      |
+----------------------+----------------------+--------------------------------+
| "--npersocket        | "--mapby ppr:<#per   | On each node, launch this many |
| <#persocket>"        | package>:package"    | processes times the number of  |
|                      |                      | processor sockets on the node. |
|                      |                      | The "--npersocket" option also |
|                      |                      | turns on the "--bindto         |
|                      |                      | socket" option. The term       |
|                      |                      | "socket" has been globally     |
|                      |                      | replaced with "package".       |
+----------------------+----------------------+--------------------------------+
| "--oversubscribe"    | "--mapby             | Nodes are allowed to be        |
|                      | :OVERSUBSCRIBE"      | oversubscribed, even on a      |
|                      |                      | managed system, and            |
|                      |                      | overloading of processing      |
|                      |                      | elements.                      |
+----------------------+----------------------+--------------------------------+
| "--pernode"          | "--mapby             | On each node, launch one       |
|                      | ppr:1:node"          | process                        |
+----------------------+----------------------+--------------------------------+
| "--ppr"              | *--mapby             | Comma-separated list of number |
|                      | ppr:<list>`*         | of processes on a given        |
|                      |                      | resource type [default:        |
|                      |                      | "none"].                       |
+----------------------+----------------------+--------------------------------+
| "--rankfile          | "--mapby rankfile:   | Use a rankfile for             |
| <FILENAME>"          | FILE=<FILENAME>"     | mapping/ranking/binding        |
+----------------------+----------------------+--------------------------------+
| "--report-bindings"  | "--display BINDINGS" | Report any bindings for        |
|                      |                      | launched processes             |
+----------------------+----------------------+--------------------------------+
| "--tag-output"       | "--output TAG"       | Tag all output with            |
|                      |                      | "[job,rank]"                   |
+----------------------+----------------------+--------------------------------+
| "--timestamp-output" | "--output TIMESTAMP" | Timestamp all application      |
|                      |                      | process output                 |
+----------------------+----------------------+--------------------------------+
| "--use-hwthread-     | "--mapby :HWTCPUS"   | Use hardware threads as        |
| cpus"                |                      | independent CPUs               |
+----------------------+----------------------+--------------------------------+
| "--xml"              | "--output XML"       | Provide all output in XML      |
|                      |                      | format                         |
+----------------------+----------------------+--------------------------------+
#
[placement-diagnostics]


Diagnostics
-----------

PRRTE provides various diagnostic reports that aid the user in
verifying and tuning the mapping/ranking/binding for a specific job.

The ":REPORT" qualifier to the "--bindto" command line option can be
used to report process bindings.

As an example, consider a node with:

* 2 processor packages,

* 4 cores per package, and

* 8 hardware threads per core.

In each of the examples below the binding is reported in a human
readable format.

   $ prun --np 4 --mapby core --bindto core:REPORT ./a.out
   [node01:103137] MCW rank 0 bound to package[0][core:0]
   [node01:103137] MCW rank 1 bound to package[0][core:1]
   [node01:103137] MCW rank 2 bound to package[0][core:2]
   [node01:103137] MCW rank 3 bound to package[0][core:3]

In the example above, processes are bound to successive cores on the
first package.

   $ prun --np 4 --mapby package --bindto package:REPORT ./a.out
   [node01:103115] MCW rank 0 bound to package[0][core:0-9]
   [node01:103115] MCW rank 1 bound to package[1][core:10-19]
   [node01:103115] MCW rank 2 bound to package[0][core:0-9]
   [node01:103115] MCW rank 3 bound to package[1][core:10-19]

In the example above, processes are bound to all cores on successive
packages in a round-robin fashion.

   $ prun --np 4 --mapby package:PE=2 --bindto core:REPORT ./a.out
   [node01:103328] MCW rank 0 bound to package[0][core:0-1]
   [node01:103328] MCW rank 1 bound to package[1][core:10-11]
   [node01:103328] MCW rank 2 bound to package[0][core:2-3]
   [node01:103328] MCW rank 3 bound to package[1][core:12-13]

The example above shows us that 2 cores have been bound per process.
The ":PE=2" qualifier states that 2 CPUs underneath the package (which
would be cores in this case) are mapped to each process.

   $ prun --np 4 --mapby core:PE=2:HWTCPUS --bindto :REPORT  hostname
   [node01:103506] MCW rank 0 bound to package[0][hwt:0-1]
   [node01:103506] MCW rank 1 bound to package[0][hwt:8-9]
   [node01:103506] MCW rank 2 bound to package[0][hwt:16-17]
   [node01:103506] MCW rank 3 bound to package[0][hwt:24-25]

The example above shows us that 2 hardware threads have been bound per
process.  In this case "prun" is directing the DVM to map by hardware
threads since we used the ":HWTCPUS" qualifier. Without that qualifier
this command would return an error since by default the DVM will not
map to resources smaller than a core.  The ":PE=2" qualifier states
that 2 processing elements underneath the core (which would be
hardware threads in this case) are mapped to each process.

   $ prun --np 4 --bindto none:REPORT  hostname
   [node01:107126] MCW rank 0 is not bound (or bound to all available processors)
   [node01:107126] MCW rank 1 is not bound (or bound to all available processors)
   [node01:107126] MCW rank 2 is not bound (or bound to all available processors)
   [node01:107126] MCW rank 3 is not bound (or bound to all available processors)

Binding is turned off in the above example, as reported.
#
[placement-fundamentals]


Fundamentals
------------

The mapping of processes to nodes can be defined not just with general
policies but also, if necessary, using arbitrary mappings that cannot
be described by a simple policy. Supported directives, given on the
command line via the "--mapby" option, include:

* "SEQ": (often accompanied by the "file=<path>" qualifier) assigns
  one process to each node specified in the file. The sequential file
  is to contain an entry for each desired process, one per line of the
  file.

* "RANKFILE": (often accompanied by the "file=<path>" qualifier)
  assigns one process to the node/resource specified in each entry of
  the file, one per line of the file.

* "DEVICE=<class|name>": assigns one process to each device in the
  node's topology, in PCI bus order.

Mapping by device is unlike the other directives in one respect worth
understanding, because it governs what binding can then do.

Every other mapping target is an object the user names directly - a
core, a NUMA domain, a package - and binding descends within it. A
device is not such an object: it hangs off the I/O side of the
topology and has no CPUs of its own. What a process is actually
placed against is the device's *locality*, meaning the nearest object
in the topology that both contains the device and has CPUs. On one
machine that may be a NUMA domain, on another a whole package,
depending on where the hardware attaches the device.

Two consequences follow, and both are deliberate:

1. Binding descends from the locality, not from the device. So
   "--mapby device=gpu --bindto core" binds each process to one core
   within the CPUs local to its GPU.

2. Asking to bind to an object larger than the locality is an error,
   not a silent widening. A binding that covers more than the device's
   locality is not "near the device" at all, which is the whole of what
   was asked for. Note this cannot be known from the command line
   alone: whether "--bindto package" is legal depends on where that
   machine attaches its devices.

PRRTE cannot restrict a process to a device the way it restricts one to
a set of CPUs - no such mechanism exists - so the assignment is only
useful if the process can find out about it. Each process is therefore
told which devices it was mapped against, as the PMIX_DEVICE_ID key of
its own proc info, which it can retrieve with:

   PMIx_Get(&myproc, PMIX_DEVICE_ID, NULL, 0, &value);

The value is ALWAYS a pmix_data_array_t of pmix_device_t, even when it
holds a single device: a process given two devices and one given a
single device are the same kind of answer differing in length, so there
is no separate single-device form to special-case.

Each entry carries the device UUID, its OS name and its type. The UUID
rather than an index is what identifies it, because a runtime's own
device numbering need not match the topology's - CUDA, for example,
orders devices by speed rather than by bus by default - so an index
would name a different device than the one PRRTE chose. The same UUID
appears in the PMIX_DEVICE_DISTANCES a process can query, which is what
lets the two be matched up.

"--display map" reports it too, as a "Device:" field on each process
line.

Where every device on a node is equally close to every CPU - which
happens when they all hang off one PCI complex rather than off
individual NUMA domains - the job still runs and each process is still
assigned its own device, but a warning is printed: the binding cannot
be made any more specific than it would have been without the
directive.

For example, using the hostfile below:

   $ cat myhostfile
   aa slots=4
   bb slots=4
   cc slots=4

The command below will launch three processes, one on each of nodes
"aa", "bb", and "cc", respectively. The slot counts don't matter; one
process is launched per line on whatever node is listed on the line.

   $ prun --hostfile myhostfile --mapby seq ./a.out

Impact of the ranking option is best illustrated by considering the
following hostfile and test cases where each node contains two
packages (each package with two cores). Using the "--mapby
ppr:2:package" option, we map two processes onto each package and
utilize the "--rank-by" option as show below:

   $ cat myhostfile
   aa
   bb

+-----------------------------------+-----------------------------------+-----------------------------------+
| Command                           | Ranks on "aa"                     | Ranks on "bb"                     |
|===================================|===================================|===================================|
| "--rankby core"                   | 0 1 ! 2 3                         | 4 5 ! 6 7                         |
+-----------------------------------+-----------------------------------+-----------------------------------+
| "--rankby package"                | 0 2 ! 1 3                         | 4 6 ! 5 7                         |
+-----------------------------------+-----------------------------------+-----------------------------------+
| "--rankby package:SPAN"           | 0 4 ! 1 5                         | 2 6 ! 3 7                         |
+-----------------------------------+-----------------------------------+-----------------------------------+

Ranking by slot provides the identical result as ranking by core in
this case — a simple progression of ranks across each node. Ranking by
package does a round-robin ranking across packages within each node
until all processes have been assigned a rank, and then progresses to
the next node.  Adding the ":SPAN" qualifier to the ranking directive
causes the ranking algorithm to treat the entire allocation as a
single entity — thus, the process ranks are assigned across all
packages before circling back around to the beginning.

The binding operation restricts the process to a subset of the CPU
resources on the node.

The processors to be used for binding can be identified in terms of
topological groupings — e.g., binding to an l3cache will bind each
process to all processors within the scope of a single L3 cache within
their assigned location. Thus, if a process is assigned by the mapper
to a certain package, then a "--bindto l3cache" directive will cause
the process to be bound to the processors that share a single L3 cache
within that package.

To help balance loads, the binding directive uses a round-robin
method, binding a process to the first available specified object type
within the object where the process was mapped. For example, consider
the case where a job is mapped to the package level, and then bound to
core. Each package will have multiple cores, so if multiple processes
are mapped to a given package, the binding algorithm will assign each
process located to a package to a unique core in a round-robin manner.

Binding can only be done to the mapped object or to a resource located
within that object.

An object is considered completely consumed when the number of
processes bound to it equals the number of CPUs within it. Unbound
processes are not considered in this computation. Additional processes
cannot be mapped to consumed objects unless the OVERLOAD qualifier is
provided via the "--bindto" command line option.

Per-App-Context Mapping
.......................

By default every application context in a job shares the same mapping,
ranking, and binding policy. In a multi-program multiple-data (MPMD)
job, each application context separated by ":" on the "prun" command
line may carry its own independent "--mapby", "--rankby", and
"--bindto" directives.

Which app such a directive describes follows one rule: the FIRST app
segment is where the command line speaks for the job. A directive
written there and nowhere else applies to the whole job, however many
apps follow. Written on any later app it describes that app alone, and
the apps that were given none take the ordinary defaults - an app that
says nothing is not agreeing with one that did.

   prun --mapby core --bindto core -n 4 app1 \
       : --mapby node --rankby fill --bindto none -n 2 app2

This launches "app1" mapped by core (processes 0-3) and "app2" mapped
by node (processes 4-5) within the same job, allowing shared-memory
and direct PMIx communication between the two apps.

   prun -n 4 app1 : --mapby node -n 2 app2

Here only "app2" is mapped by node; "app1" is placed by the default
rules.

Every mapping policy may be given per app - "seq", "rankfile",
"ppr:N:obj" and "pe-list=..." included - so two apps of one job may be
placed by two different mapping components. "--display devel-map"
names the component that placed each app.

Qualifiers that describe the whole job - the following govern the job
as a whole, so wherever they are written they are applied to the job:

* "OVERSUBSCRIBE" / "NOOVERSUBSCRIBE": oversubscription applies to
  all apps sharing the same nodes.

* "INHERIT" / "NOINHERIT": whether a spawned child job inherits the
  parent's policies is a job-wide property.

Apps that say nothing about these are silent, not dissenting. Only
apps that answer the same question in OPPOSITE ways are refused, and
that aborts the job with an error.

The "NOLOCAL" qualifier IS per app context. It prevents that app's
processes from running on the head node without affecting other apps
in the same job:

   prun --mapby slot:nolocal -n 8 app1 : --mapby slot -n 1 app2

Here "app1" avoids the head node; "app2" may run anywhere.

Per-app directives can also be supplied via the "PMIx_Spawn" API by
placing "PMIX_MAPBY", "PMIX_RANKBY", "PMIX_BINDTO" and "PMIX_PPR" keys
in the per-app "info[]" array on the corresponding "pmix_app_t". On that
path there is no "first app" rule: the array a key was written in says
what it describes.

Note that "PMIX_MAPPER" is NOT supported, per job or per app, and a
spawn request carrying it is refused. Naming a mapping component says
nothing that "PMIX_MAPBY" has not already said - the mapping policy is
what selects the component - and the two can contradict each other.
Describe the placement you want with "PMIX_MAPBY" and let PRRTE choose
the component that performs it.

Global rank assignment remains contiguous across all apps regardless
of per-app ranking directives: ranks are assigned in app-context order
starting from 0, so the first app receives ranks 0..N-1 and each
subsequent app continues from the next unassigned rank. The per-app
"--rankby" directive controls only the ordering of ranks within that
app. The same holds for the mappers that number their own processes: a
"rankfile" or "seq" file given to an app numbers THAT app's ranks, and
PRRTE offsets them into the job's numbering.

Default process mapping/ranking/binding policies can also be set with
MCA parameters, overridden by the command line options when provided.
MCA parameters can be set on the "prte" command line when starting the
DVM (or in the "prterun" command line for a single-execution job), but
also in a system or user "mca-params.conf" file or as environment
variables, as described in the MCA section below. Some examples
include:

+-----------------------------------+-------------------+-----------+
| "prun" option                     | MCA parameter key |   Value   |
|===================================|===================|===========|
| "--mapby core"                    |       "mapby"     | "core"    |
+-----------------------------------+-------------------+-----------+
| "--mapby package"                 |       "mapby"     | "package" |
+-----------------------------------+-------------------+-----------+
| "--rankby core"                   |       "rankby"    | "core"    |
+-----------------------------------+-------------------+-----------+
| "--bindto core"                   |       "bindto"    | "core`"   |
+-----------------------------------+-------------------+-----------+
| "--bindto package"                |       "bindto"    | "package" |
+-----------------------------------+-------------------+-----------+
| "--bindto none"                   |       "bindto"    | "none"    |
+-----------------------------------+-------------------+-----------+
#
[placement-limits]


Overloading and Oversubscribing
-------------------------------

This section explores the difference between the terms "overloading"
and "oversubscribing". Users are often confused by the difference
between these two scenarios. As such, this section provides a number
of scenarios to help illustrate the differences.

* "--mapby :OVERSUBSCRIBE" allow more processes on a node than
  allocated

* "--bindto <object>:overload-allowed" allows for binding more than
  one process in relation to a CPU

The important thing to remember with *oversubscribing* is that it can
be defined separately from the actual number of CPUs on a node. This
allows the mapper to place more or fewer processes per node than CPUs.
By default, PRRTE uses cores to determine slots in the absence of such
information provided in the hostfile or by the resource manager
(except in the case of the "--host" as described in the section on
that command line option.

The important thing to remember with *overloading* is that it is
defined as binding more processes than CPUs. By default, PRRTE uses
cores as a means of counting the number of CPUs. However, the user can
adjust this. For example when using the ":HWTCPUS" qualifier to the "
--mapby" option PRRTE will use hardware threads as a means of
counting the number of CPUs.

For the following examples consider a node with:

* 2 processor packages,

* 10 cores per package, and

* 8 hardware threads per core.

Consider the node from above with the hostfile below:

   $ cat myhostfile
   node01 slots=32
   node02 slots=32

The "slots" token tells PRRTE that it can place up to 32 processes
before *oversubscribing* the node.

If we run the following:

   prun --np 34 --hostfile myhostfile --mapby core
       --bindto core hostname

It will return an error at the binding time indicating an
*overloading* scenario.

The mapping mechanism assigns 32 processes to "node01" matching the
"slots" specification in the hostfile. The binding mechanism will bind
the first 20 processes to unique cores leaving it with 12 processes
that it cannot bind without overloading one of the cores (putting more
than one process on the core).

Using the "overload-allowed" qualifier to the "--bindto core" option
tells PRRTE that it may assign more than one process to a core.

If we run the following:

   prun --np 34 --hostfile myhostfile --mapby core
       --bindto core:overload-allowed hostname

This will run correctly placing 32 processes on "node01", and 2
processes on "node02". On "node01" two processes are bound to cores
0-11 accounting for the overloading of those cores.

Alternatively, we could use hardware threads to give binding a lower
level CPU to bind to without overloading.

If we run the following:

   prun --np 34 --hostfile myhostfile --mapby core:HWTCPUS
       --bindto hwthread hostname

This will run correctly placing 32 processes on "node01", and 2
processes on "node02". On "node01" two processes are mapped to cores
0-11 but bound to different hardware threads on those cores (the
logical first and second hardware thread). Thus no hardware threads
are overloaded at binding time.

In both of the examples above the node is not oversubscribed at
mapping time because the hostfile set the oversubscription limit to
"slots=32" for each node. It is only after we exceed that limit that
PRRTE will throw an oversubscription error.

Consider next if we ran the following:

   prun --np 66 --hostfile myhostfile
       --mapby core:HWTCPUS --bindto hwthread hostname

This will return an error at mapping time indicating an
oversubscription scenario. The mapping mechanism will assign all of
the available slots (64 across 2 nodes) and be left two processes to
map. The only way to map those processes is to exceed the number of
available slots putting the job into an oversubscription scenario.

You can force PRRTE to oversubscribe the nodes by using the
":OVERSUBSCRIBE" qualifier to the "--mapby" option as seen in the
example below:

   prun --np 66 --hostfile myhostfile
       --mapby core:HWTCPUS:OVERSUBSCRIBE --bindto hwthread hostname

This will run correctly placing 34 processes on "node01" and 32 on
"node02".  Each process is bound to a unique hardware thread.


Overloading vs. Oversubscription: Package Example
=================================================

Let's extend these examples by considering the package level. Consider
the same node as before, but with the hostfile below:

   $ cat myhostfile
   node01 slots=22
   node02 slots=22

The lowest level CPUs are "cores" and we have 20 total (10 per
package).

If we run:

   prun --np 20 --hostfile myhostfile --mapby package
       --bindto package:REPORT hostname

Then 10 processes are mapped to each package, and bound at the package
level.  This is not overloading since we have 10 CPUs (cores)
available in the package at the hardware level.

However, if we run:

   prun --np 21 --hostfile myhostfile --mapby package
       --bindto package:REPORT hostname

Then 11 processes are mapped to the first package and 10 to the second
package.  At binding time we have an overloading scenario because
there are only 10 CPUs (cores) available in the package at the
hardware level. So the first package is overloaded.


Overloading vs. Oversubscription: Hardware Threads Example
==========================================================

Similarly, if we consider hardware threads.

Consider the same node as before, but with the hostfile below:

   $ cat myhostfile
   node01 slots=165
   node02 slots=165

The lowest level CPUs are "hwthreads" (because we are going to use the
":HWTCPUS" qualifier) and we have 160 total (80 per package).

If we re-run (from the package example) and add the ":HWTCPUS"
qualifier:

   prun --np 21 --hostfile myhostfile --mapby package:HWTCPUS
       --bindto package:REPORT hostname

Without the ":HWTCPUS" qualifier this would be overloading (as we saw
previously). The mapper places 11 processes on the first package and
10 to the second package. The processes are still bound to the package
level. However, with the ":HWTCPUS" qualifier, it is not overloading
since we have 80 CPUs (hwthreads) available in the package at the
hardware level.

Alternatively, if we run:

   prun --np 161 --hostfile myhostfile --mapby package:HWTCPUS
       --bindto package:REPORT hostname

Then 81 processes are mapped to the first package and 80 to the second
package.  At binding time we have an overloading scenario because
there are only 80 CPUs (hwthreads) available in the package at the
hardware level.  So the first package is overloaded.
