Running applications on Windows

Running HP-MPI from CCP

There are two ways to run HP-MPI: command line and scheduler GUI. Both approaches can be used to access the functionality of the scheduler. The command line scheduler options are similar to the GUI options.

The following instructions are in the context of the GUI, but equivalent command line options are also listed.

Microsoft Compute Cluster Pack job scheduler uses the term 'job' to refer to an allocation of resources, while a 'task' is command that is scheduled to run using a portion of a job allocation. It is important that HP-MPI’s mpirun be submitted as a task that uses only a single processor. This allows the remaining resources within the job to be used for the creation of the remaining application ranks (and daemons). This is different from MSMPI, which requires that all of the processors in the job be allocated to the MSMPI mpiexec task.

Figure 3-1 “Job Allocation” shows the relationship between HP-MPI processes and the job allocation. A single task, "Task 1" is submitted and assigned a single CPU resource inside a larger job allocation. This task contains the mpirun command. Solid lines show the creation of local daemons and ranks using standard process creation calls. The creation of remote ranks are handled by mpirun by creating additional tasks within the job allocation. Only the task which starts mpirun is submitted by the user.

Figure 3-1 Job Allocation

To run an MPI application, submit the mpirun command to the scheduler. HP-MPI uses the environment of the task and job where mpirun is executing to launch the required mpids that start the ranks.

It is important that mpirun uses only a single processor for its task within the job so the resources can be used by the other processes within the MPI application. (This is the opposite of MSMPI, which requires all of the processor to be allocated to the mpiexec task by the MPI application.)

Submitting jobs

The section includes general information for submitting jobs either from the GUI or the command line.

As part of the mpirun task submitted, the following flags are commonly used with mpirun:

Verify that rank.exe is on a shared directory.

Included in the help directory is an example template (.xml file). To use this template, change the processor count on the Submit Jobs window, Processors tab, and edit the Tasks command to include flags and the rank executable. The job will run with ranks being equal to the number of processors, or the number of nodes if the -nodex flag is used.

Submitting jobs from the CCS GUI

To execute an HP-MPI job from the CCS GUI:

For convenience, generic templates can be created and saved using Save As Template in the Submit Job window.

Running HP-MPI from command line on CCP systems

To perform the same steps via command line, execute 3 commands:

For example:

> job new /jobname:[example job]/numprocessors:12 ^
/projectname:HPMPI
Job Queued, ID: 242

This will create a job resource and return a jobid, but not submit it.

> job add 242 /stdout:"\\shared\dir\%CCP_JOBNAME%-^
%CCPTASKCONTEXT%.out"/stderr:"\\shared\dir\%CCP_JOBNAME%-^
%CCPTASKCONTEXT%.err""%MPI_ROOT%\bin\mpirun" -ccp -prot ^ -netaddr 172.16.150.20/24 -TCP \\shared\dir\rank.exe -arg1 ^ -arg2

> job submit /id:242

Automatic job submittal

HP-MPI has added functionality which will automatically create, schedule, and submit an mpirun command to the scheduler from the command line. Although this functionality is limited, it is the easiest way to quickly start mpirun.

To have HP-MPI submit the mpirun to the CCP scheduler, type your mpirun command from the command prompt, and include the -ccp flag. mpirun will detect it is not running in a CCP job and will automatically submit the mpirun command to the scheduler. -np is required to indicate the size of the job to schedule.

Flags that have special meaning when doing so are as follows:

-np N

Indicates the number of ranks to execute. This is a required flag for automatic job submittal.

-hostlist <quoted-host-list>

Indicates what nodes to use for the job.

-e MPI_WORKDIR=<directory>

Used to indicate the working directory. Default is the current directory.

When submitting a job, mpirun will set the job’s 'working directory' to the current directory, or MPI_WORKDIR if provided, with the assumption the resulting directory name will be valid across the entire cluster. It will construct the stdout and stderr filenames from the root name of the rank, and append the jobid and taskid to the filenames, with the extensions .out and .err respectively.

The following is an example of submitting a job via the automatic submittal:

C:\> Documents and Settings\smith> ^ "%MPI_ROOT%\bin\mpirun.exe" -ccp -np 6 ^ \\share\directory\smith\HelloWorld.exe
mpirun: Submitting job to scheduler and exiting
Submitting job to ccp scheduler on this node
mpirun: Drive is not a network mapped - using local drive.
mpirun: HPMPI Job 1116 submitted to cluster mpiccp1

This will schedule and run 6 ranks of HelloWorld.exe. Standard output and standard error are placed in the current directory, 'HelloWorld-1116.1.out' and 'HelloWorld-1116.1.err' respectively.

The following example changes the directory to a share drive, and uses the current directory as the work directory for the submitted job:

C:\> Documents and Settings\smith>s:
S:\> cd smith
S:\smith> "%MPI_ROOT%\bin\mpirun.exe" -ccp -np 6 ^
-hostlist mpiccp1,mpiccp2 HelloWorld.exe
mpirun: Submitting job to scheduler and exiting
Submitting job to ccp scheduler on this node
mpirun: HPMPI Job 1117 submitted to cluster mpiccp1

Here the 'S:' drive is interpreted as the mapped network drive. The rank HelloWorld.exe is located in the current directory, and the stdout and stderr files are placed in the current working directory.

Running on CCP with an appfile

mpirun jobs submitted to CCP can run using appfile mode. The resources in the appfile must match the allocated job resources for this to run correctly.

The -ccp flag will generate an appfile that uses the allocated CCP job resources as the machines and rank counts in the appfile, then launches the mpirun job using this appfile.

If the user wishes to provide their own appfile, the tools described below have been provided.

The executable mpi_nodes.exe (located in "%MPI_ROOT%\bin") has been provided which will return the job resources, in the same format as the CCP_NODES environment variable. Alternately, the user could write an appfile, then select the exact resources needed from the Submit jobs window, then the Advanced tab as needed. But this defeats the purpose of the job scheduler because the CCP_NODES environment variable only lists the resources allocated to the task, not the job.

<Node-Count> [<Node> <Processors-on-node>] ...

The script submission_script.vbs (found in "%MPI_ROOT%\help") is an example of using mpi_nodes.exe to generate an appfile, and submit the mpirun command.

There are many other ways to accomplish the same thing. Other scripting languages can be used to convert the output of mpi_nodes.exe into the appropriate appfile.

Or a script using all the job resources and the CCP_NODES environment variable can construct the appropriate appfile, then submit a single processor mpirun task to CCP_JOBID and exit. Here the mpirun task will be queued up with no available processors until the script (using all job processors) finishes, then start executing.

	NOTE: If using this method, don't forget to include /stdout and /stderr options to the CCP job add command when adding the mpirun task.

Again, there are many different ways to generate your own appfile for use by mpirun. In all cases, the mpirun command that is launched will look like a 'normal' HP-MPI appfile launch:

mpirun -f generated-appfile [other HP-MPI options]

Refer to “More information about appfile runs”.