By default, the make command invokes the GNU make program. GNU make has the ability to make independent targets concurrently. For example, if building a program requires compiling 10 source files, and the compilations can be done independently, make can manage multiple compilations at once — the number of jobs is user selectable. More precisely, each target's rules are run normally (sequentially within the rule). Typically the rules for an object file target is a single compilation line, so it is common to talk about concurrent compilations, though GNU make is more general.
On non-cluster platforms or command nodes, matching concurrency to the number of cores often works well. It also often works well to specify a few more jobs than cores so that one job can proceed while another is waiting for I/O. On an HP XC system, there is the potential to use compute nodes to do compilations, and there are a variety of ways to make this happen.
One way is to prefix the actual compilation line in the rule with an srun command. So, instead of executing cc foo.c -o foo.o it would execute srun cc foo.c -o foo.o. With concurrency, multiple command nodes would have multiple srun commands instead of multiple cc commands. For projects that recursively run make on subdirectories, the recursive make can be run on the compute nodes. For example:
$ cd subdir; srun $(MAKE)... |
Further, if the recursive make is run remotely, it can be told to use concurrency on the remote node. For example:
$ cd subdir; srun -n1 -N1 $(MAKE) -j4... |
This can cause multiple makes to run concurrently, each building their targets concurrently. The -N1 option is used to reserve the entire node, because it is intended to be used for multiple compilations. The following examples illustrate these ideas. In GNU make, a $(VARIABLE) that is unspecified is replaced with nothing. Therefore, not specifying PREFIX keeps the original makefile's behavior, but specifying PREFIX to appropriate srun command prefixes will cause concurrency within the build.
For more information about GNU parallel make, refer to the make manpage. For additional sources of GNU information, refer to the references provided in “About This Document”.
In this section, three different ways to parallelize a make procedure are illustrated. The smg98 package is used to illustrate these three procedures. The smg98 package is available at the following URL:
http://www.llnl.gov/asci/applications/SMG98README.html
These procedures take advantage of the GNU make -j switch, which specifies the number of jobs to run simultaneously. Refer to the make manpage for more information about this switch.
The following parallel make approaches are described:
Example Procedure 1 — Go through the directories serially and have the make procedure within each directory be parallel. (Modified makefile: Makefile_type1).
Example Procedure 2 — Go through the directories in parallel and have the make procedure within each directory be serial (Modified makefile: Makefile_type2).
Example Procedure 3 — Go through the directories in parallel and have the make procedure within each directory be parallel (Modified makefile: Makefile_type3).
The original makefile is shown below:
#BHEADER***********************************************************
# (c) 1998 The Regents of the University of California
#
# See the file COPYRIGHT_and_DISCLAIMER for a complete copyright
# notice, contact person, and disclaimer.
#
# $Revision: 1.1 $
#EHEADER***********************************************************
SHELL = /bin/sh
srcdir = .
HYPRE_DIRS =\
utilities\
struct_matrix_vector\
struct_linear_solvers\
test
all:
@ \
for i in ${HYPRE_DIRS}; \
do \
if [ -d $$i ]; \
then \
echo "Making $$i ..."; \
(cd $$i; make); \
echo ""; \
fi; \
done
clean:
@ \
for i in ${HYPRE_DIRS}; \
do \
if [ -d $$i ]; \
then \
echo "Cleaning $$i ..."; \
(cd $$i; make clean); \
fi; \
done
veryclean:
@ \
for i in ${HYPRE_DIRS}; \
do \
if [ -d $$i ]; \
then \
echo "Very-cleaning $$i ..."; \
(cd $$i; make veryclean); \
fi; \
done |
 |
Example Procedure 1 |
|
Go through the directories serially and have the make procedure within each directory be parallel.
For the purpose of this exercise we are only parallelizing the “make all” component. The “clean” and “veryclean” components can be parallelized in a similar fashion.
Modified makefile:
all:
@ \
for i ${HYPRE_DIRS}; \
do \
if [ -d $$i ]; \
then \
echo "Making $$i ..."; \
echo $(PREFIX) $(MAKE) $(MAKE_J) -C $$i; \
$(PREFIX) $(MAKE) $(MAKE_J) -C $$i; \
fi; \
done |
By modifying the makefile to reflect the changes illustrated above, we will now be processing each directory serially and parallelize the individual makes within each directory. The modified Makefile is invoked as follows:
$ make PREFIX=’srun –n1 –N1 MAKE_J='-j4' |
Example Procedure 2 |
|
Go through the directories in parallel and have the make procedure within each directory be serial.
For the purpose of this exercise we are only parallelizing the “make all” component. The “clean” and “veryclean” components can be parallelized in a similar fashion.
Modified makefile:
all:
$(MAKE) $(MAKE_J) struct_matrix_vector/libHYPRE_mv.a
struct_linear_solvers/libHYPRE_ls.a utilities/libHYPRE_utilities.a
$(PREFIX) $(MAKE) -C test
struct_matrix_vector/libHYPRE_mv.a:
$(PREFIX) $(MAKE) -C struct_matrix_vector
struct_linear_solvers/libHYPRE_ls.a:
$(PREFIX) $(MAKE) -C struct_linear_solvers
utilities/libHYPRE_utilities.a:
$(PREFIX) $(MAKE) -C utilities |
The modified Makefile is invoked as follows:
$ make PREFIX='srun -n1 -N1' MAKE_J='-j4' |
Example Procedure 3 |
|
Go through the directories in parallel and have the make procedure within each directory be parallel. For the purpose of this exercise, we are only parallelizing the “make all” component. The “clean” and “veryclean” components can be parallelized in a similar fashion.
Modified makefile:
all:
$(MAKE) $(MAKE_J) struct_matrix_vector/libHYPRE_mv.a
struct_linear_solvers/libHYPRE_ls.a utilities/libHYPRE_utilities.a
$(PREFIX) $(MAKE) $(MAKE_J) -C test
struct_matrix_vector/libHYPRE_mv.a:
$(PREFIX) $(MAKE) $(MAKE_J) -C struct_matrix_vector
struct_linear_solvers/libHYPRE_ls.a:
$(PREFIX) $(MAKE) $(MAKE_J) -C struct_linear_solvers
utilities/libHYPRE_utilities.a:
$(PREFIX) $(MAKE) $(MAKE_J) -C utilities |
The modified Makefile is invoked as follows:
$ make PREFIX='srun -n1 -N1' MAKE_J='-j4' |
Printable version
