# -*- text -*-
#
# Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
#                         University Research and Technology
#                         Corporation.  All rights reserved.
# Copyright (c) 2004-2005 The University of Tennessee and The University
#                         of Tennessee Research Foundation.  All rights
#                         reserved.
# Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
#                         University of Stuttgart.  All rights reserved.
# Copyright (c) 2004-2005 The Regents of the University of California.
#                         All rights reserved.
# Copyright (c) 2011-2020 Cisco Systems, Inc.  All rights reserved
# Copyright (c) 2011      Los Alamos National Security, LLC.
#                         All rights reserved.
# Copyright (c) 2014-2020 Intel, Inc.  All rights reserved.
# Copyright (c) 2021-2026 Nanook Consulting  All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
# This is the US/English general help file for PRRTE's prun.
#
[map-by]

NOTE: PRRTE accepts both the new "--mapby" and the older
deprecated "--map-by" cmd line options. For simplicity, the
following description will refer to the new "--mapby" form.

Processes are mapped based on one of the following directives as
applied at the job level:

* "SLOT" assigns procs to each node up to the number of available
  slots on that node before moving to the next node in the allocation

* "HWTHREAD" assigns a proc to each hardware thread on a node in a
  round-robin manner up to the number of available slots on that node
  before moving to the next node in the allocation

* "CORE" (default) assigns a proc to each core on a node in a round-
  robin manner up to the number of available slots on that node before
  moving to the next node in the allocation

* "L1CACHE" assigns a proc to each L1 cache on a node in a round-robin
  manner up to the number of available slots on that node before
  moving to the next node in the allocation

* "L2CACHE" assigns a proc to each L2 cache on a node in a round-robin
  manner up to the number of available slots on that node before
  moving to the next node in the allocation

* "L3CACHE" assigns a proc to each L3 cache on a node in a round-robin
  manner up to the number of available slots on that node before
  moving to the next node in the allocation

* "NUMA" assigns a proc to each NUMA region on a node in a round-robin
  manner up to the number of available slots on that node before
  moving to the next node in the allocation

* "PACKAGE" assigns a proc to each package on a node in a round-robin
  manner up to the number of available slots on that node before
  moving to the next node in the allocation

* "NODE" assigns processes in a round-robin fashion to all nodes in
  the allocation, with the number assigned to each node capped by the
  number of available slots on that node

* "SEQ" (often accompanied by the file=<path> qualifier) assigns one
  process to each node specified in the file. The sequential file is
  to contain an entry for each desired process, one per line of the
  file.

* "PPR:N":resource maps N procs to each instance of the specified
  resource type in the allocation

* "RANKFILE" (often accompanied by the file=<path> qualifier) assigns
  one process to the node/resource specified in each entry of the
  file, one per line of the file.

* "PE-LIST=a,b" assigns procs to each node in the allocation based on
  the ORDERED qualifier. The list is comprised of comma-delimited
  ranges of CPUs to use for this job. If the ORDERED qualifier is not
  provided, then each node will be assigned procs up to the number of
  available slots, capped by the total number of specified processes and
  availability of the specified CPUs (i.e. the number of procs cannot exceed
  the number of specified CPUs), with each proc bound to all the
  specified CPUs.
  If ORDERED is given, then one proc will be assigned to each of the
  specified CPUs, if available, capped by the number of slots on each
  node and the total number of specified processes. Providing the
  OVERLOAD qualifier to the "bind-to" option removes the check on
  availability of the CPU in both cases.

Any directive can include qualifiers by adding a colon (":") and any
combination of one or more of the following (delimited by colons) to
the "--mapby" option (except where noted):

* "PE=n" bind n CPUs to each process (can not be used in combination
  with rankfile or pe-list directives)

* "SPAN" load balance the processes across the allocation by treating
  the allocation as a single "super-node" (can not be used in
  combination with "slot", "node", "seq", "ppr", "rankfile", or "pe-
  list" directives)

* "OVERSUBSCRIBE" allow more processes on a node than processing
  elements. NOTE: this describes the whole job. It may be written in a
  per-app "--mapby" string, where it is taken to describe the job (see
  "Per-App-Context Mapping" below).

* "NOOVERSUBSCRIBE" means "!OVERSUBSCRIBE". NOTE: describes the whole
  job (see above).

* "NOLOCAL" do not launch processes on the same node as "prun". This
  qualifier may be applied per app context in an MPMD job — see
  "Per-App-Context Mapping" below.

* "HWTCPUS" use hardware threads as CPU slots

* "CORECPUS" use cores as CPU slots (default)

* "INHERIT" indicates that a child job (i.e., one spawned from within
  an application) shall inherit the placement policies of the parent
  job that spawned it. NOTE: this describes the whole job; written in a
  per-app "--mapby" string it is taken to describe the job.

* "NOINHERIT" means "!INHERIT". NOTE: describes the whole job (see
  above).

* "FILE=<path>" (path to file containing sequential or rankfile
  entries).

* "ORDERED" only applies to the "PE-LIST" option to indicate that
  procs are to be bound to each of the specified CPUs in the order in
  which they are assigned (i.e., the first proc on a node shall be
  bound to the first CPU in the list, the second proc shall be bound
  to the second CPU, etc.)

Note:

  Directives and qualifiers are case-insensitive and can be shortened
  to the minimum number of characters to uniquely identify them. Thus,
  "L1CACHE" can be given as "l1cache" or simply as "L1".

The type of CPU (core vs hwthread) used in the mapping algorithm is
determined as follows:

* by user directive on the command line via the HWTCPUS qualifier to
  the "--mapby" directive

* by setting the "rmaps_default_mapping_policy" MCA parameter to
  include the "HWTCPUS" qualifier. This parameter sets the default
  value for a PRRTE DVM — qualifiers are carried across to DVM jobs
  started via "prun" unless overridden by the user's command line

* defaults to CORE in topologies where core CPUs are defined, and to
  hwthreads otherwise.

If your application uses threads, then you probably want to ensure
that you are either not bound at all (by specifying "--bind-to none"),
or bound to multiple cores using an appropriate binding level or
specific number of processing elements per application process via the
"PE=#" qualifier to the "--mapby" command line directive.

Per-App-Context Mapping (MPMD Jobs)
------------------------------------

In a multi-program multiple-data (MPMD) job, each application context
separated by ":" on the "prun" command line may carry its own
"--mapby", "--rankby", and "--bindto" directives.

Which app such a directive describes follows one rule: the FIRST app
segment is where the command line speaks for the job. A directive
written there and nowhere else applies to the whole job, however many
apps follow. A directive written on any later app describes that app
alone; apps that were given none take the ordinary defaults, since an
app that says nothing is not agreeing with one that did.

Examples:

  prun --mapby core -n 4 app1 : -n 2 app2

Both apps are mapped by core: the only directive is on the first app,
so it describes the job.

  prun --mapby core -n 4 app1 : --mapby node --rankby fill -n 2 app2

Here "app1" is mapped by core and "app2" is mapped by node with fill
ranking. The binding policy defaults to the job-level value for both.

  prun -n 4 app1 : --mapby node -n 2 app2

Only "app2" is mapped by node; "app1" takes the default mapping.

  prun --mapby slot:nolocal -n 8 app1 : --mapby slot -n 1 app2

Here "app1" is excluded from the head node while "app2" may run
anywhere, including the head node.

Every mapping policy may be given per app - "seq", "rankfile",
"ppr:N:obj" and "pe-list=..." included, each with the file or pattern
it needs - so two apps of one job may be placed by two different
mapping components. "--display devel-map" names the component that
placed each app.

Qualifiers that describe the whole job:

* "OVERSUBSCRIBE" / "NOOVERSUBSCRIBE" and "INHERIT" / "NOINHERIT"
  describe the job, not an app, so wherever they are written they are
  applied to the job. Apps that say nothing about them are silent, not
  dissenting; only apps that answer the same question in OPPOSITE ways
  are refused, and that aborts the job with an error.

* "NOLOCAL" MAY be applied per app. It prevents that specific app's
  processes from running on the head node without affecting other apps.

Per-app directives can also be supplied via the "PMIx_Spawn" API by
placing "PMIX_MAPBY", "PMIX_RANKBY", "PMIX_BINDTO" and "PMIX_PPR" keys
in the per-app "info[]" array on the corresponding "pmix_app_t". On that
path there is no "first app" rule: the array a key was written in says
what it describes.

Note that "PMIX_MAPPER" is NOT supported, per job or per app, and a
spawn request carrying it is refused. Naming a mapping component says
nothing that "PMIX_MAPBY" has not already said - the mapping policy is
what selects the component - and the two can contradict each other.
Describe the placement you want with "PMIX_MAPBY" and let PRRTE choose
the component that performs it.

A more detailed description of the mapping, ranking, and binding
procedure can be obtained via the "--help placement" option.
#
[mapby]

#include#help-mapby.txt#map-by
