To run MPI jobs on a set of nodes that have a mix of PCI-X and PCI Express InfiniBand boards, you must include the following option with the HP-MPI mpirun command:
-e MPI_IB_MTU=1024