Using counter instrumentation

Counter instrumentation is a lightweight method for generating cumulative runtime statistics for your MPI applications. When you create an instrumentation profile, HP-MPI creates an output file in ASCII format.

You can create instrumentation profiles for applications linked with the standard HP-MPI library. For applications linked with HP-MPI version 2.1 or later, you can also create profiles for applications linked with the thread-compliant library (-lmtmpi). Instrumentation is not supported for applications linked with the diagnostic library (-ldmpi).

Creating an instrumentation profile

Counter instrumentation is a lightweight method for generating cumulative runtime statistics for MPI applications. When you create an instrumentation profile, HP-MPI creates an ASCII format file containing statistics about the execution.

Instrumentation is not supported for applications linked with the diagnostic library (-ldmpi).

The syntax for creating an instrumentation profile is:

mpirun -i prefix[:l][:nc][:off]

where

prefix: Specifies the instrumentation output file prefix. The rank zero process writes the application’s measurement data to prefix.instr in ASCII. If the prefix does not represent an absolute pathname, the instrumentation output file is opened in the working directory of the rank zero process when MPI_Init is called.
l: Locks ranks to CPUs and uses the CPU’s cycle counter for less invasive timing. If used with gang scheduling, the :l is ignored.
nc: Specifies no clobber. If the instrumentation output file exists, MPI_Init aborts.
off: Specifies counter instrumentation is initially turned off and only begins after all processes collectively call MPIHP_Trace_on.

For example, to create an instrumentation profile for an executable called compute_pi:

% $MPI_ROOT/bin/mpirun -i compute_pi -np 2 compute_pi

This invocation creates an ASCII file named compute_pi.instr containing instrumentation profiling.

Although -i is the preferred method of controlling instrumentation, the same functionality is also accessible by setting the MPI_INSTR environment variable. Refer to “MPI_INSTR” for syntax information.

Specifications you make using mpirun -i override any specifications you make using the MPI_INSTR environment variable.

MPIHP_Trace_on and MPIHP_Trace_off

By default, the entire application is profiled from MPI_Init to MPI_Finalize. However, HP-MPI provides the nonstandard MPIHP_Trace_on and MPIHP_Trace_off routines to collect profile information for selected code sections only.

To use this functionality:

Insert the MPIHP_Trace_on and MPIHP_Trace_off pair around code that you want to profile.
Build the application and invoke mpirun with the -i <prefix> off; option. -i <index> off; specifies that counter instrumentation is enabled but initially turned off (refer to “mpirun ” and “MPI_INSTR”). Data collection begins after all processes collectively call MPIHP_Trace_on. HP-MPI collects profiling information only for code between MPIHP_Trace_on and MPIHP_Trace_off

Viewing ASCII instrumentation data

The ASCII instrumentation profile is a text file with the .instr extension. For example, to view the instrumentation file for the compute_pi.f application, you can print the prefix.instr file. If you defined prefix for the file as compute_pi, as you did when you created the instrumentation file in “Creating an instrumentation profile ”, you would print compute_pi.instr.

The ASCII instrumentation profile provides the version, the date your application ran, and summarizes information according to application, rank, and routines. Figure 4-1 “ASCII instrumentation profile” is an example of an ASCII instrumentation profile.

The information available in the prefix.instr file includes:

Overhead time—The time a process or routine spends inside MPI. For example, the time a process spends doing message packing or spinning waiting for message arrival.
Blocking time—The time a process or routine is blocked waiting for a message to arrive before resuming execution.




	NOTE: Overhead and blocking times are most useful when using `-e MPI_FLAGS=y0`.

Communication hot spots—The processes in your application between which the largest amount of time is spent in communication.
Message bin—The range of message sizes in bytes. The instrumentation profile reports the number of messages according to message length.

Figure 4-1 “ASCII instrumentation profile” displays the contents of the example report compute_pi.instr.

Figure 4-1 ASCII instrumentation profile

Version: HP MPI 01.08.00.00 B6060BA - HP-UX 11.0
Date:    Mon Apr 01 15:59:10 2002
Processes: 2
User time:   6.57%
MPI time :  93.43% [Overhead:93.43% Blocking:0.00%]
-----------------------------------------------------------------
--------------------   Instrumentation Data --------------------
-----------------------------------------------------------------
Application Summary by Rank (second):
Rank       Proc CPU Time     User Portion       System Portion
-----------------------------------------------------------------
0      0.040000        0.010000( 25.00%)     0.030000( 75.00%)
1      0.030000        0.010000( 33.33%)     0.020000( 66.67%)
-----------------------------------------------------------------
Rank       Proc Wall Time      User                  MPI
----------------------------------------------------------------
0       0.126335     0.008332(  6.60%)       0.118003( 93.40%)
1       0.126355     0.008260(  6.54%)       0.118095( 93.46%)
-----------------------------------------------------------------
Rank       Proc MPI Time      Overhead              Blocking
-----------------------------------------------------------------
0        0.118003       0.118003(100.00%)    0.000000(  0.00%)
1        0.118095       0.118095(100.00%)    0.000000(  0.00%)
-----------------------------------------------------------------
Routine Summary by Rank:
Rank Routine   Statistic   Calls   Overhead(ms)   Blocking(ms)
--------------------------------------------------------------
0
MPI_Bcast                    1        5.397081        0.000000
MPI_Finalize                 1        1.238942        0.000000
MPI_Init                     1      107.195973        0.000000
MPI_Reduce                   1        4.171014        0.000000
--------------------------------------------------------------
1
MPI_Bcast                    1        5.388021        0.000000
MPI_Finalize                 1        1.325965        0.000000
MPI_Init                     1      107.228994        0.000000
MPI_Reduce                   1        4.152060        0.000000
--------------------------------------------------------------
Message Summary by Rank Pair:
SRank    DRank  Messages (minsize,maxsize)/[bin]    Totalbytes
--------------------------------------------------------------
0
             1      1          (4, 4)                  4
                    1          [0..64]                 4
--------------------------------------------------------------
1
             0      1          (8, 8)                  8
                    1         [0..64]                  8
--------------------------------------------------------------

Using counter instrumentation

Technical documentation

» Table of Contents

» Glossary

» Index

Creating an instrumentation profile

MPIHP_Trace_on and MPIHP_Trace_off

Viewing ASCII instrumentation data