MPI options

Options for LSF users

These options launch ranks as they are in appfile mode on the hosts specified in the environment variable.

Options for prun users

Options for SLURM users

MPIRUN_OPTIONS is a mechanism for specifying additional command line arguments to mpirun. If this environment variable is set, then any mpirun command will behave as if the arguments in MPIRUN_OPTIONS had been specified on the mpirun command line. For example:

% export MPIRUN_OPTIONS="-v -prot"

% $MPI_ROOT/bin/mpirun -np 2 /path/to/program.x

would be equivalent to running

% $MPI_ROOT/bin/mpirun -v -prot -np 2 /path/to/program.x

When settings are supplied on the command line, in the MPIRUN_OPTIONS variable, and in an hpmpi.conf file, the resulting command line is as if the hpmpi.conf settings had appeared first, followed by the MPIRUN_OPTIONS, followed by the actual command line. And since the settings are parsed left to right, this means the later settings have higher precedence than the earlier ones.

MPI_FLAGS modifies the general behavior of HP-MPI. The MPI_FLAGS syntax is a comma separated list as follows:

[edde,][exdb,][egdb,][eadb,][ewdb,][l,][f,][i,] [s[a|p][#],][y[#],][o,][+E2,][C,][D,][E,][T,][z]

where

MPI_MT_FLAGS controls runtime options when you use the thread-compliant version of HP-MPI. The MPI_MT_FLAGS syntax is a comma separated list as follows:

[ct,][single,][fun,][serial,][mult]

where

Setting MPI_MT_FLAGS=ct has the same effect as setting MPI_FLAGS=s[a][p]#, when the value of # that is greater than 0. MPI_MT_FLAGS=ct takes priority over the default MPI_FLAGS=sp0 setting. Refer to “MPI_FLAGS”.

The single, fun, serial, and mult options are mutually exclusive. For example, if you specify the serial and mult options in MPI_MT_FLAGS, only the last option specified is processed (in this case, the mult option). If no runtime option is specified, the default is mult.

For more information about using MPI_MT_FLAGS with the
thread-compliant library, refer to “Thread-compliant library”.

MPI_ROOT indicates the location of the HP-MPI tree. If you move the HP-MPI installation directory from its default location in /opt/mpi for HP-UX and /opt/hpmpi for Linux, set the MPI_ROOT environment variable to point to the new location. See “Directory structure for HP-UX and Linux” for more information.

MPI_WORKDIR changes the execution directory. This variable is ignored when srun or prun is used.

MPI_BIND_MAP allows specification of the integer CPU numbers, ldom numbers, or CPU masks. These are a list of integers separated by commas (,).

MPI_CPU_AFFINITY is an alternative method to using -cpu_bind on the command line for specifying binding strategy. The possible settings are LL, RANK, MAP_CPU, MASK_CPU, LDOM, CYCLIC, BLOCK, RR, FILL, PACKED, SLURM, and MAP_LDOM.

MPI_CPU_SPIN allows selection of spin value. The default is 2 seconds.

MPI_FLUSH_FCACHE clears the file-cache (buffer-cache). Add "-e MPI_FLUSH_FCACHE[=x]" to the mpirun command line and the file-cache will be flushed before the code starts; where =x is an optional percent of memory at which to flush. If the memory in the file-cache is greater than x, the memory is flushed. The default value is 0 (in which case a flush is always performed). Only the lowest rank# on each host flushes the file-cache; limited to one flush per host/job.

Setting this environment variable saves time if, for example, the file-cache is currently using 8% of the memory and =x is set to 10. In this case, no flush is performed.

Example output:

MPI_FLUSH_FCACHE set to 0, fcache pct = 22, attempting to flush fcache on host opteron2

MPI_FLUSH_FCACHE set to 10, fcache pct = 3, no fcache flush required on host opteron2

Memory is allocated with mmap, then munmap'd afterwards.

MP_GANG enables gang scheduling on HP-UX systems only. Gang scheduling improves the latency for synchronization by ensuring that all runable processes in a gang are scheduled simultaneously. Processes waiting at a barrier, for example, do not have to wait for processes that are not currently scheduled. This proves most beneficial for applications with frequent synchronization operations. Applications with infrequent synchronization, however, may perform better if gang scheduling is disabled.

Process priorities for gangs are managed identically to timeshare policies. The timeshare priority scheduler determines when to schedule a gang for execution. While it is likely that scheduling a gang will preempt one or more higher priority timeshare processes, the gang-schedule policy is fair overall. In addition, gangs are scheduled for a single time slice, which is the same for all processes in the system.

MPI processes are allocated statically at the beginning of execution. As an MPI process creates new threads, they are all added to the same gang if MP_GANG is enabled.

The MP_GANG syntax is as follows:

[ON|OFF]

where

For multihost configurations, you need to set MP_GANG for each appfile entry. Refer to the -e option in “Creating an appfile”.

You can also use the HP-UX utility mpsched to enable gang scheduling. Refer to the HP-UX gang_sched and mpsched man pages for more information.

	NOTE: The MP_GANG feature will be deprecated in a future release.

Point-to-point bcopy() is disabled by setting MPI_2BCOPY to 1. Valid on PA-RISC only.

MPI_MAX_WINDOW is used for one-sided applications. It specifies the maximum number of windows a rank can have at the same time. It tells HP-MPI to allocate enough table entries. The default is 5.

% export MPI_MAX_WINDOW=10

The above example allows 10 windows to be established for one-sided communication.

MPI_DLIB_FLAGS controls runtime options when you use the diagnostics library. The MPI_DLIB_FLAGS syntax is a comma separated list as follows:

[ns,][h,][strict,][nmsg,][nwarn,][dump:prefix,]
[dumpf:prefix][xNUM]

where

Refer to “Using the diagnostics library” for more information.

On PA-RISC systems, a stack trace is printed when the following signals occur within an application:

In the event one of these signals is not caught by a user signal handler, HP-MPI will display a brief stack trace that can be used to locate the signal in the code.

Signal 10: bus error
PROCEDURE TRACEBACK:

(0)   0x0000489c   bar + 0xc   [././a.out]
(1)   0x000048c4   foo + 0x1c   [,/,/a.out]
(2)   0x000049d4   main + 0xa4   [././a.out]
(3)   0xc013750c   _start + 0xa8   [/usr/lib/libc.2]
(4)   0x0003b50         $START$ + 0x1a0   [././a.out]                                                

This feature can be disabled for an individual signal handler by declaring a user-level signal handler for the signal. To disable for all signals, set the environment variable MPI_NOBACKTRACE:

% setenv MPI_NOBACKTRACE

See “Backtrace functionality” for more information.

MPI_INSTR enables counter instrumentation for profiling HP-MPI applications. The MPI_INSTR syntax is a colon-separated list (no spaces between options) as follows:

prefix[:l][:nc][:off]

where

Refer to “Using counter instrumentation” for more information.

Even though you can specify profiling options through the MPI_INSTR environment variable, the recommended approach is to use the mpirun command with the -i option instead. Using mpirun to specify profiling options guarantees that multihost applications do profiling in a consistent manner. Refer to “mpirun ” for more information.

Counter instrumentation and trace-file generation are mutually exclusive profiling techniques.

	NOTE: When you enable instrumentation for multihost runs, and invoke mpirun either on a host where at least one MPI process is running, or on a host remote from all your MPI processes, HP-MPI writes the instrumentation output file (prefix.instr) to the working directory on the host that is running rank 0.

When you use the TotalView debugger, HP-MPI uses your PATH variable to find TotalView. You can also set the absolute path and TotalView specific options in the TOTALVIEW environment variable. This environment variable is used by mpirun.

% setenv TOTALVIEW /opt/totalview/bin/totalview

MPI_IC_ORDER is an environment variable whose default contents are "ibv:vapi:udapl:psm:mx:gm:elan:itapi:TCP" and instructs HP-MPI to search in a specific order for the presence of an interconnect. Lowercase selections imply use if detected, otherwise keep searching. An uppercase option demands that the interconnect option be used, and if it cannot be selected the application will terminate with an error. For example:

% export MPI_IC_ORDER="ibv:vapi:udapl:psm:mx:gm:elan: \
itapi:TCP"

% export MPIRUN_OPTIONS="-prot"

% $MPI_ROOT/bin/mpirun -srun -n4 ./a.out

The command line for the above will appear to mpirun as $MPI_ROOT/bin/mpirun -prot -srun -n4 ./a.out and the interconnect decision will look for the presence of Elan and use it if found. Otherwise, interconnects will be tried in the order specified by MPI_IC_ORDER.

The following is an example of using TCP over GigE, assuming GigE is installed and 192.168.1.1 corresponds to the ethernet interface with GigE. Note the implicit use of -netaddr 192.168.1.1 is required to effectively get TCP over the proper subnet.

% export MPI_IC_ORDER="ibv:vapi:udapl:psm:mx:gm:elan: \
itapi:TCP"

% export MPIRUN_SYSTEM_OPTIONS="-netaddr 192.168.1.1"

% $MPI_ROOT/bin/mpirun -prot -TCP -srun -n4 ./a.out

On an XC system, the cluster installation will define the MPI interconnect search order based on what is present on the system.

When HP-MPI is determining the availability of a given interconnect on Linux, it tries to open libraries and find loaded modules based on a collection of variables of the form

This is described in more detail in “Interconnect support”.

The use of interconnect environment variables MPI_ICLIB_ELAN, MPI_ICLIB_GM, MPI_ICLIB_ITAPI, MPI_ICLIB_MX, MPI_ICLIB_UDAPL, MPI_ICLIB_VAPI, and MPI_ICLIB_VAPIDIR has been deprecated. Refer to “Interconnect support” for more information on interconnect environment variables.

MPI_COMMD routes all off-host communication through daemons rather than between processes. The MPI_COMMD syntax is as follows:

out_frags,in_frags

where

Only works with the -commd option. When -commd is used, MPI_COMMD specifies daemon communication fragments.

Defines mapping of ranks to IB cards.

% setenv MPI_IB_CARD_ORDER <card#>[:port#]

Where:

Card:port can be a comma separated list which drives the assignment of ranks to cards and ports within the cards.

Note that HP-MPI numbers the ports on a card from 0 to N-1, whereas utilities such as vstat display ports numbered 1 to N.

Examples:

To use the 2nd IB card:

% mpirun -e MPI_IB_CARD_ORDER=1 ...

To use the 2nd port of the 2nd card:

% mpirun -e MPI_IB_CARD_ORDER=1:1 ...

To use the 1st IB card:

% mpirun -e MPI_IB_CARD_ORDER=0 ...

To assign ranks to multiple cards:

% mpirun -e MPI_IB_CARD_ORDER=0,1,2
will assign the local ranks per node in order to each card.

% mpirun -hostlist "host0 4 host1 4"
creates ranks 0-3 on host0 and ranks 4-7 on host1. Will assign rank 0 to card 0, rank 1 to card 1, rank 2 to card 2, rank 3 to card 0 all on host0. And will assign rank 4 to card 0, rank 5 to card 1, rank 6 to card 2, rank 7 to card 0 all on host1.

% mpirun -hostlist -np 8 "host0 host1"
creates ranks 0 through 7 alternatingly on host0, host1, host0, host1, etc. Will assign rank 0 to card 0, rank 2 to card 1, rank 4 to card 2, rank 6 to card 0 all on host0. And will assign rank 1 to card 0, rank 3 to card 1, rank 5 to card 2, rank 7 to card 0 all on host1.

HP-MPI supports IB partitioning via Mellanox VAPI and OpenFabrics Verbs API.

By default, HP-MPI will search the unique full membership partition key from the port partition key table used. If no such pkey is found, an error is issued. If multiple pkeys are found, all such pkeys are printed and an error message is issued.

If the environment variable MPI_IB_PKEY has been set to a value, either in hex or decimal, the value is treated as the pkey, and the pkey table is searched for the same pkey. If the pkey is not found, an error message is issued.

When a rank selects a pkey to use, a check is made to make sure all ranks are using the same pkey. If ranks are not using the same pkey, and error message is issued.

MPI_IBV_QPPARAMS=a,b,c,d,e Specifies QP settings for IBV where:

MPI_VAPI_QPPARAMS=a,b,c,d specifies time-out setting for VAPI where:

MPI_GLOBMEMSIZE=e Where e is the total bytes of shared memory of the job. If the job size is N, then each rank has e/N bytes of shared memory. 12.5% is used as generic. 87.5% is used as fragments. The only way to change this ratio is to use MPI_SHMEMCNTL.

Set MPI_NO_MALLOCLIB to avoid using HP-MPI’s ptmalloc implementation and instead use the standard libc implementation (or perhaps a malloc implementation contained in the application).

See “Improved deregistration via ptmalloc (Linux only)” for more information.

MPI_PAGE_ALIGN_MEM causes the HP-MPI library to page align and page pad memory. This is for multi-threaded InfiniBand support.

% export MPI_PAGE_ALIGN_MEM=1

MPI_PHYSICAL_MEMORY allows the user to specify the amount of physical memory in kilobytes available on the system. MPI normally attempts to determine the amount of physical memory for the purpose of determining how much memory to pin for RDMA message transfers on InfiniBand and Myrinet GM. The value determined by HP-MPI can be displayed using the -dd option. If HP-MPI specifies an incorrect value for physical memory, this environment variable can be used to specify the value explicitly:

% export MPI_PHYSICAL_MEMORY=1048576

The above example specifies that the system has 1GB of physical memory.

MPI_PIN_PERCENTAGE and MPI_PHYSICAL_MEMORY are ignored unless InfiniBand or Myrinet GM is in use.

MPI_RANKMEMSIZE=d Where d is the total bytes of shared memory of the rank. Specifies the shared memory for each rank. 12.5% is used as generic. 87.5% is used as fragments. The only way to change this ratio is to use MPI_SHMEMCNTL. MPI_RANKMEMSIZE differs from MPI_GLOBMEMSIZE, which is the total shared memory across all the ranks on the host. MPI_RANKMEMSIZE takes precedence over MPI_GLOBMEMSIZE if both are set. Both MPI_RANKMEMSIZE and MPI_GLOBMEMSIZE are mutually exclusive to MPI_SMEMCNTL. If MPI_SHMEMCNTL is set, then the user cannot set the other two, and vice versa.

MPI_PIN_PERCENTAGE communicates the maximum percentage of physical memory (see MPI_PHYSICAL_MEMORY) that can be pinned at any time. The default is 20%.

% export MPI_PIN_PERCENTAGE=30

The above example permits the HP-MPI library to pin (lock in memory) up to 30% of physical memory. The pinned memory is shared between ranks of the host that were started as part of the same mpirun invocation. Running multiple MPI applications on the same host can cumulatively cause more than one application’s MPI_PIN_PERCENTAGE to be pinned. Increasing MPI_PIN_PERCENTAGE can improve communication performance for communication intensive applications in which nodes send and receive multiple large messages at a time, such as is common with collective operations. Increasing MPI_PIN_PERCENTAGE allows more large messages to be progressed in parallel using RDMA transfers, however pinning too much of physical memory may negatively impact computation performance. MPI_PIN_PERCENTAGE and MPI_PHYSICAL_MEMORY are ignored unless InfiniBand or Myrinet GM is in use.

MPI_SHMEMCNTL controls the subdivision of each process’s shared memory for the purposes of point-to-point and collective communications. It cannot be used in conjunction with MPI_GLOBMEMSIZE. The MPI_SHMEMCNTL syntax is a comma separated list as follows:

nenv, frag, generic

where

MPI_SHMEMCNTL=a,b,c where:

Instructs the underlying malloc implementation to avoid mmaps and instead use sbrk() to get all the memory used. The default is MPI_USE_MALLOPT_AVOID_MMAP=0.

MPI_LOCALIP specifies the host IP address that is assigned throughout a session. Ordinarily, mpirun determines the IP address of the host it is running on by calling gethostbyaddr. However, when a host uses a SLIP or PPP protocol, the host’s IP address is dynamically assigned only when the network connection is established. In this case, gethostbyaddr may not return the correct IP address.

The MPI_LOCALIP syntax is as follows:

xxx.xxx.xxx.xxx

where xxx.xxx.xxx.xxx specifies the host IP address.

MPI_MAX_REMSH=N HP-MPI includes a startup scalability enhancement when using the -f option to mpirun. This enhancement allows a large number of HP-MPI daemons (mpid) to be created without requiring mpirun to maintain a large number of remote shell connections.

When running with a very large number of nodes, the number of remote shells normally required to start all of the daemons can exhaust the available file descriptors. To create the necessary daemons, mpirun uses the remote shell specified with MPI_REMSH to create up to 20 daemons only, by default. This number can be changed using the environment variable MPI_MAX_REMSH. When the number of daemons required is greater than MPI_MAX_REMSH, mpirun will create only MPI_MAX_REMSH number of remote daemons directly. The directly created daemons will then create the remaining daemons using an n-ary tree, where n is the value of MPI_MAX_REMSH. Although this process is generally transparent to the user, the new startup requires that each node in the cluster is able to use the specified MPI_REMSH command (e.g. rsh, ssh) to each node in the cluster without a password. The value of MPI_MAX_REMSH is used on a per-world basis. Therefore, applications which spawn a large number of worlds may need to use a small value for MPI_MAX_REMSH. MPI_MAX_REMSH is only relevant when using the -f option to mpirun. The default value is 20.

Allows control of the selection process for TCP/IP connections. The same functionality can be accessed by using the -netaddr option to mpirun. See “mpirun options” for more information.

By default, HP-MPI attempts to use ssh on Linux and remsh on HP-UX. On Linux, we recommend that ssh users set StrictHostKeyChecking=no in their ~/.ssh/config.

To use rsh on Linux instead, the following script needs to be run as root on each node in the cluster:

% /opt/hpmpi/etc/mpi.remsh.default

Or, to use rsh on Linux, use the alternative method of manually populating the files /etc/profile.d/hpmpi.csh and /etc/profile.d/hpmpi.sh with the following settings respectively:

setenv MPI_REMSH rsh

export MPI_REMSH=rsh

On HP-UX, MPI_REMSH specifies a command other than the default remsh to start remote processes. The mpirun, mpijob, and mpiclean utilities support MPI_REMSH. For example, you can set the environment variable to use a secure shell:

% setenv MPI_REMSH /bin/ssh

HP-MPI allows users to specify the remote execution tool to use when HP-MPI needs to start processes on remote hosts. The tool specified must have a call interface similar to that of the standard utilities: rsh, remsh and ssh. An alternate remote execution tool, such as ssh, can be used on HP-UX by setting the environment variable MPI_REMSH to the name or full path of the tool to use:

% export MPI_REMSH=ssh

% $MPI_ROOT/bin/mpirun <options> -f <appfile>

HP-MPI also supports setting MPI_REMSH using the -e option to mpirun:

% $MPI_ROOT/bin/mpirun -e MPI_REMSH=ssh <options> -f \ <appfile>

This release also supports setting MPI_REMSH to a command which includes additional arguments:

% $MPI_ROOT/bin/mpirun -e MPI_REMSH="ssh -x" <options> \
-f <appfile>

When using ssh, first ensure that it is possible to use ssh from the host where mpirun is executed to the other nodes without ssh requiring any interaction from the user.

-e MPI_RDMA_INTRALEN=262144 Specifies the size (in bytes) of the transition from shared memory to interconnect when -intra=mix is used. For messages less than or equal to the specified size, shared memory will be used. For messages greater than that size, the interconnect will be used. TCP/IP, Elan, MX, and PSM do not have mixed mode.

MPI_RDMA_MSGSIZE=a,b,c Specifies message protocol length where:

MPI_RDMA_NENVELOPE=N Specifies the number of short message envelope pairs for each connection if RDMA protocol is used, where N is the number of envelope pairs. The default is between 8 and 128 depending on the number of ranks.

MPI_RDMA_NFRAGMENT=N Specifies the number of long message fragments that can be concurrently pinned down for each process, either sending or receiving. The max number of fragments that can be pinned down for a process is 2*N. The default value of N is 128.

MPI_RDMA_NONESIDED=N Specifies the number of one-sided operations that can be posted concurrently for each rank, no matter the destination. The default is 8.

MPI_RDMA_NSRQRECV=K Specifies the number of receiving buffers used when the shared receiving queue is used, where K is the number of receiving buffers. If N is the number of offhost connection from a rank, then the default value can be calculated as:

The smaller of the values Nx8 and 2048.

In the above example, the number of receiving buffers are calculated as 8 times the number of offhost connections. If this number is greater than 2048, then 2048 is used as the maximum number.

Allows prun options to be implicitly added to the launch command when SPAWN functionality is used to create new ranks with prun.

Allows srun options to be implicitly added to the launch command when SPAWN functionality is used to create new ranks with srun.

Allows additional srun options to be specified such as --label.

% setenv MPI_SRUNOPTIONS <option>

HP-MPI provides the capability to automatically assume that prun is the default launching mechanism. This mode of operation automatically classifies arguments into 'prun' and 'mpirun' arguments and correctly places them on the command line.The assumed prun mode also allows appfiles to be interpreted for command line arguments and translated into prun mode. The implied prun method of launching is useful for applications which embed or generate their mpirun invocations deeply within the application.

See Appendix C for more information.

Provides an easy way to modify the arguments contained in an appfile by supplying a list of space-separated arguments that mpirun should ignore.

% setenv MPI_USEPRUN_IGNORE_ARGS <option>

HP-MPI provides the capability to automatically assume that srun is the default launching mechanism. This mode of operation automatically classifies arguments into 'srun' and 'mpirun' arguments and correctly places them on the command line.The assumed srun mode also allows appfiles to be interpreted for command line arguments and translated into srun mode. The implied srun method of launching is useful for applications which embed or generate their mpirun invocations deeply within the application. This allows existing ports of an application from an HP-MPI supported platform to XC.

See Appendix C for more information.

Provides an easy way to modify the arguments contained in an appfile by supplying a list of space-separated arguments that mpirun should ignore.

% setenv MPI_USESRUN_IGNORE_ARGS <option>

In the example below, the command line contains a reference to -stdio=bnone which is filtered out because it is set in the ignore list.

% setenv MPI_USESRUN_VERBOSE 1

% setenv MPI_USESRUN_IGNORE_ARGS -stdio=bnone

% setenv MPI_USESRUN 1

% setenv MPI_SRUNOPTION --label

% bsub -I -n4 -ext "SLURM[nodes=4]" \
$MPI_ROOT/bin/mpirun -stdio=bnone -f appfile -- pingpong

Job <369848> is submitted to default queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on lsfhost.localdomain>>
/opt/hpmpi/bin/mpirun
unset
MPI_USESRUN;/opt/hpmpi/bin/mpirun
-srun  ./pallas.x -npmin 4 pingpong

Allows prun specific options to be added automatically to the mpirun command line. For example:

% export MPI_PRUNOPTIONS="-m cyclic -x host0"

% mpirun -prot -prun -n2 ./a.out

is equivalent to:

% mpirun -prot -prun -m cyclic -x host0 -n2 ./a.out

The integer value indicates the number of simultaneous messages larger than 16KB that may be transmitted to a single rank at once via TCP/IP. Setting this variable to a larger value can allow HP-MPI to utilize more parallelism during its low-level message transfers, but can greatly reduce performance by causing switch congestion. Setting MPI_TCP_CORECVLIMIT to zero will not limit the number of simultaneous messages a rank may receive at once. The default value is 0.

Specifies, in bytes, the amount of system buffer space to allocate for sockets when using the TCP/IP protocol for communication. Setting MPI_SOCKBUFSIZE results in calls to setsockopt (..., SOL_SOCKET, SO_SNDBUF, ...) and setsockopt (..., SOL_SOCKET, SO_RCVBUF, ...). If unspecified, the system default (which on many systems is 87380 bytes) is used.

By default when Elan is in use, the HP-MPI library uses Elan’s native collective operations for performing MPI_Bcast and MPI_ Barrier operations on MPI_COMM_WORLD sized communicators. This behavior can be changed by setting MPI_USE_LIBELAN to “false” or “0”, in which case these operations will be implemented using point-to-point Elan messages.

To turn off:

% export MPI_USE_LIBELAN=0

The use of Elan’s native collective operations may be extended to include communicators which are smaller than MPI_COMM_WORLD by setting the MPI_USE_LIBELAN_SUB environment variable to a positive integer. By default, this functionality is disabled due to the fact that libelan memory resources are consumed and may eventually cause runtime failures when too many sub-communicators are created.

% export MPI_USE_LIBELAN_SUB=10

By default, HP-MPI only provides exclusive window locks via Elan lock when using the Elan interconnect. In order to use HP-MPI shared window locks, the user must turn off Elan lock and use window locks via shared memory. In this way, both exclusive and shared locks are from shared memory. To turn off Elan locks, set MPI_ELANLOCK to zero.

% export MPI_ELANLOCK=0