Improved deregistration via ptmalloc (Linux only)

To achieve the best performance on RDMA enabled interconnects like InfiniBand and Myrinet, the MPI library must be aware when memory is returned to the system in malloc() and free() calls. To enable more robust handling of that information, HP-MPI contains a copy of the ptmalloc implementation and uses it by default.

For applications with particular needs, there are a number of available modifications to this default configuration. To avoid using HP-MPI's ptmalloc implementation and instead use the standard libc implementation (or perhaps a malloc implementation contained in the application), set the environment variable MPI_NO_MALLOCLIB at runtime.

If the above option is applied so that the ptmalloc contained in HP-MPI is not used, then there is a risk of MPI not being informed of when memory is returned to the system. This can be alleviated with the settings MPI_USE_MALLOPT_SBRK_PROTECTION and MPI_USE_MALLOPT_AVOID_MMAP at runtime, which essentially results in the libc malloc implementation not returning memory to the system.

There are cases where these two settings cannot keep libc from returning memory to the system, in particular when multiple threads call malloc/free at the same time. In these cases, the only remaining option is to disable HP-MPI's lazy deregistration by giving the -ndd flag to mpirun.

Note that in the default case where the ptmalloc contained within HP-MPI is used, the above cases are all avoided and lazy deregistration works correctly as is. So the above tunables are only recommended for applications with special requirements concerning their malloc/free usage.