Administering LSF-HPC

LSF-HPC dispatches all jobs locally. The default installation of LSF-HPC for SLURM on the HP XC system provides a job starter script that is configured for use by all LSF-HPC queues. This job starter script adjusts the LSB_HOSTS and LSB_MCPU_HOSTS environment variables to the correct resource values in the allocation. Then, the job starter script uses the srun command to launch the user task on the first node in the allocation.

If this job starter script is not configured for a queue, the user jobs begin execution locally on the LSF-HPC execution host. In this case, it is recommended that the user job uses one or more srun commands to make use of the resources allocated to the job. Work done on the LSF-HPC execution host competes for core time with the LSF-HPC daemons, and could affect the overall performance of LSF-HPC on the HP XC system.

The bqueues -l command displays the full queue configuration, including whether or not a job starter script has been configured. See the Platform LSF documentation or the bqueues(1) manpage for more information on the use of this command.

For example, consider an LSF-HPC configuration in which node n20 is the LSF-HPC execution host and nodes n[1-10] are in the SLURM lsf partition. The default normal queue contains the job starter script, but the unscripted queue does not have the job starter script configured.

Example 13-2 Comparison of Queues and the Configuration of the Job Starter Script

$ bqueues -l normal | grep JOB_STARTER JOB_STARTER: /opt/hptc/lsf/bin/job_starter.sh $ bqueues -l unscripted | grep JOB_STARTER JOB_STARTER: $ bsub -Is hostname Job <66> is submitted to the default queue <normal>. <<Waiting for dispatch...>> <<Starting on lsfhost.localdomain>> n10 $ bsub -Is -q unscripted hostname Job <67> is submitted to the default queue <unscripted>. <<Waiting for dispatch...>> <<Starting on lsfhost.localdomain>> n20

This release of the HP XC System Software provides an LSF-HPC queue JOB_STARTER script, which is configured for all default queues during HP XC installation. This JOB_STARTER script performs three tasks:

The LSB_HOSTS and LSB_MCPU_HOSTS environment variables, as initially established by LSF-HPC, do not accurately reflect the host names of the HP XC system nodes that SLURM allocated for the user's job. This JOB_STARTER script corrects these environment variables so that existing applications compatible with LSF can use them without further adjustment.

The SLURM srun command used by the JOB_STARTER script ensures that every interactive job submitted by a user begins on the first allocated node. Without the JOB_STARTER script, all interactive user jobs would start on the LSF-HPC execution host. This behavior is not consistent with batch job submissions or Standard LSF behavior in general, and creates the potential for a bottleneck in performance as both the LSF-HPC daemons and local user tasks compete for processor cycles.

The JOB_STARTER script has one drawback: all interactive I/O runs through the srun command in the JOB_STARTER script. This means full tty support is not available for interactive sessions, resulting in no prompting when a shell is launched. The workaround is to set your display to support launching an xterm instead of a shell.

The JOB_STARTER script is located at /opt/hptc/lsf/bin/job_starter.sh, and is preconfigured for all of the queues created during the default LSF-HPC installation on the HP XC system. HP recommends that you configure the JOB_STARTER script for all queues.

To disable the JOB_STARTER script, simply remove it or comment it out from the lsb.queues configuration file. For more information on the JOB_STARTER option and configuring queues, see Administering Platform LSF on the HP XC Documentation CD.

For more information on configuring JOB_STARTER scripts and how they work, see the Standard LSF documentation

The integration of LSF-HPC with SLURM includes the addition of a SLURM-based external scheduler. Users can submit SLURM parameters in the context of their jobs. This enables users to make specific topology-based allocation requests. See the HP XC System Software User's Guide for more information.

An lsf partition is created in SLURM; this partition contains all the nodes that LSF-HPC manages. This partition must be configured such that only the superuser can make allocation requests (RootOnly=YES). This configuration prevents other users from directly accessing the resources that are being managed by LSF-HPC. The LSF-HPC daemons, running as the superuser, make allocation requests on behalf of the owner of the job to be dispatched. This is how LSF-HPC creates SLURM allocations for users' jobs to be run.

The lsf partition must be configured so that the nodes can be shared by default (Shared=FORCE). Thus, LSF-HPC can allocate serial jobs by different users on a per-processor basis (rather than on a per-node basis) by default, which makes the best use of the resources. This setting also enables LSF-HPC to support preemption by allowing a new job to run while an existing job is suspended on the same resource.

SLURM nodes can be in various states. Table 13-1 describes how LSF-HPC interprets each node state.

LSF-HPC failover is of critical concern because only one node in the HP XC system runs the LSF-HPC daemons. During installation, you select the primary LSF execution host from the nodes on the HP XC system that have the resource management role; although that node could also be a compute node, it is not recommended. Other nodes that also have the resource management role are designated as potential LSF execution host backups.

To address this concern, LSF-HPC is configured on HP XC with a virtual host name (vhost) and a virtual IP (vIP). The virtual IP and host name are used because they can be moved from one node to another, and maintain a consistent LSF interface. By default, the virtual IP is an internal IP on the HP XC administration network, and the virtual host name is lsfhost.localdomain. The LSF execution host is configured to host the vIP, then the LSF-HPC daemons are started on that node.

The Nagios infrastructure contains a module that monitors the LSF-HPC virtual IP. If it detects a problem with the virtual IP (for example, the inability to ping it), the monitoring code assumes the node is down and chooses a new LSF execution host from the backup candidate nodes on which to set up the virtual IP and restart LSF-HPC.

LSF-HPC is configured to start up automatically when the HP XC system starts up, through the use of the /etc/init.d/lsf script.

If LSF-HPC stops running, you can start it with the controllsf command, as shown here:

# controllsf start

This command searches through a list of nodes with the lsf service until it finds a node to run LSF-HPC.

Alternatively, you can invoke the following command to start LSF-HPC on the current node:

# controllsf start here

At system shutdown, the /etc/init.d/lsf script ensures an orderly shutdown of LSF-HPC.

You can use the controllsf command, as shown here, to stop LSF-HPC regardless of where it is active in the HP XC system:

# controllsf stop

	Note: At least two nodes must have the resource management roles to enable LSF-HPC failover. One is selected as the master (primary LSF execution host), and the others are considered backup nodes. At any time, LSF-HPC daemons start and run only on the master node.

The Nagios LSF-HPC failover module monitors the virtual IP associated with the primary LSF execution host. When LSF-HPC failover is enabled on the HP XC system, and if the primary LSF execution host fails, the Nagios LSF-HPC failover module detects that the node is unresponsive and initiates failover:

LSF-HPC monitoring and failover are implemented on the HP XC system as tools that prepare the environment for the LSF-HPC execution host daemons on a given node, start the daemons, then watch the node to ensure that it remains active.

After a standard installation, the HP XC system is initially configured so that:

The only direct interaction between LSF-HPC and the LSF-HPC monitoring and failover tools occurs at LSF-HPC startup, when the daemons are started in the virtual environment, and at failover, when the existing daemons are shut down cleanly before the virtual environment is moved to a new host.

You have the option of enabling or disabling LSF-HPC failover at any time. For more information, see controllsf(8).

LSF-HPC and SLURM are managed independently; one is not critically affected if the other goes down.

SLURM has no dependency on LSF-HPC.

LSF-HPC needs SLURM to schedule jobs. If SLURM becomes unresponsive, LSF-HPC drops its processor count to 1 and closes the HP XC virtual host. When SLURM is available again, LSF-HPC adjusts its processor count accordingly and reopens the host.

You assign nodes to both SLURM and LSF-HPC by assigning them the resource management role; this role includes both the lsf and slurm_controller services.

By default, the HP XC resource management system attempts to place the SLURM controller and the LSF execution host on the same node to constrain the use of system resources. If only one node has the resource management role, the LSF-HPC execution daemons and the SLURM control daemon both run on that node.

If two nodes are assigned the resource management role, by default, the first node becomes the primary resource management node, and the second node is the backup resource management node.

If more than two nodes are assigned the resource management role, the first becomes the primary resource management host and the second becomes the backup SLURM host and the first LSF-HPC failover candidate. Additional nodes with the resource management role can serve as LSF-HPC failover nodes if the either or both of the first two nodes are down.

Resource management candidate nodes are ordered in ASCII sort order by node name, after the head node, which is taken first.

Example

In this example, nodes n3, n4, n15, and n16 are resource management nodes, and n16 is the head node. The selection list is ordered as shown, and the nodes have the corresponding assignments:

You can use the controllsf command to change these assignments.

You can also change the selection of the primary and backup nodes for the SLURM control daemon by editing the SLURM configuration file, /hptc_cluster/slurm/etc/slurm.conf.

In the event of an LSF-HPC failover, LSF-HPC terminates each job that was previously running. These jobs finish with an exit code of 122.

LSF-HPC cannot monitor the running jobs to determine if the job is running appropriately or if it is hung indefinitely when the HP XC LSF execution host fails and the LSF-HPC daemons are restarted on another node.

Ensure that each LSF-HPC queue configured in the lsb.queues file includes 122 as a requeue exit value so that these jobs will be requeued and rerun. At a minimum, the entry for each queue should resemble the following:

REQUEUE_EXIT_VALUES=122

Should the node hosting LSF-HPC becomes unresponsive, the Nagios check_lsf plug-in takes action.

Table 13-2 lists the Nagios messages for LSF failover monitor status:

Table 13-3 describes the environment variables in the lsf.conf file that you can use to enhance LSF-HPC.

Table 13-3  Environment Variables for LSF-HPC Enhancement (lsf.conf File)

Environment Variable	Description
LSB_RLA_PORT=port_number	This entry specifies the TCP port used for communication between the LSF-HPC allocation adapter (RLA) and the SLURM scheduler plug-in. The default port number is 6883.
LSB_RLA_TIMEOUT=seconds	This entry defines the communications timeout between RLA and its clients (for example, sbatchd and the SLURM scheduler plug-in.) The default value is 10 seconds.
LSB_RLA_UPDATE=seconds	This entry specifies how often the LSF-HPC scheduler refreshes free node information from RLA. The default value is 600 seconds.
LSB_RLA_WORKDIR=directory	This entry specifies the directory to store the RLA status file. It enables RLA to recover its original state when it restarts. When RLA first starts, it creates the directory defined by LSB_RLA_WORKDIR if it does not exist, then creates subdirectories for each host. Avoid using /tmp or any other directory that is automatically cleaned up by the system. Unless your installation has restrictions on the LSB_SHAREDIR directory, use the default for LSB_SHAREDIR. The default value is LSB_SHAREDIR/cluster_name/rla_workdir.
LSB_SLURM_BESTFIT=Y	This setting provides for either of two systemwide scheduling modes: first-fit Under this mode, the scheduler chooses the first-met free nodes to allocate. By default, HPC SLURM integration works under first-fit mode. best-fit Under this mode, the scheduler applies a set of criteria to choose nodes with minimal capacities that satisfy the job request. You can set LSB_SLURM_BESTFIT=Y to switch the scheduler to work under best-fit mode. In a heterogeneous HP XC system, a best-fit allocation may be preferable for clusters running a mix of serial and parallel jobs. In this context, best fit means: "the nodes that minimally satisfy the requirements." Nodes with the maximum number of processors are chosen first. For parallel and serial jobs, the nodes with minimal memory, minimal tmp space, and minimal weight are chosen.
LSB_SLURM_NONLSF_USE=Y	LSF-HPC is configured to terminate unrecognized jobs in the SLURM lsf partition periodically. This entry stops LSF-HPC from periodically terminating unrecognized jobs in the SLURM lsf partition.
LSF_ENABLE_EXTSCHEDULER=Y|y	This setting enables external scheduling for LSF-HPC The default value is Y, which is automatically set by hpcinstall.
LSF_HPC_EXTENSIONS="ext_name,..."	This setting enables Platform LSF-HPC extensions. This setting is undefined by default. The following extension names are supported: SHORT_EVENTFILE This compresses long host name lists when event records are written to the lsb.events and lsb.acct files for large parallel jobs. The short host string has the format: number_of_hosts*real_host_name When SHORT_EVENTFILE is enabled, older daemons and commands (pre-LSF Version 6.1) cannot recognize the lsb.acct and lsb.events file format. For example, the original host list record is as follows: 6 "hostA" "hostA" "hostA" "hostA" "hostB" "hostC" Redundant host names are removed and the host count is changed so that the short host list record is as follows: 3 "4*hostA" "hostB" "hostC" When LSF_HPC_EXTENSION="SHORT_EVENTFILE" is set, and LSF reads the host list from the lsb.events or lsb.acct files, the compressed host list is expanded into a normal host list. This setting applies to the following events: JOB_START — when a normal job is dispatched. JOB_FORCE — when a job is forced with the brun command. JOB_CHUNK — when a job is inserted into a job chunk. JOB_FORWARD — when a job is forwarded to a MultiCluster leased host. JOB_FINISH in lsb.acct. SHORT_PIDLIST This shortens the output from the bjobs command to eliminate many of the process IDs (PIDs) for a job. The bjobs command displays only the first ID and a count of the process group IDs (PGIDs) and process IDs for the job. Without the SHORT_PIDLIST setting, the bjobs -l command displays all the PGIDs and PIDs for the job. With SHORT_PIDLIST set, the bjobs -l command displays a count of the PGIDs and PIDs. RESERVE_BY_STARTTIME LSF selects the reservation that gives the job the earliest predicted start time. By default, if multiple host groups are available for reservation, LSF chooses the largest possible reservation based on the number of slots. When backfill is configured, this can lead to larger jobs not running as their start times are pushed further into the future. BRUN_WITH_TOPOLOGY If a topology request can be satisfied for a brun job, brun preserves the topology request. LSF allocates the resource according to the request and tries to run the job with the requested topology. If allocation fails because the topology request cannot be satisfied, the job is requeued. By default, the job topology request is ignored by the scheduler when it creates an allocation if BRUN_WITH_TOPOLOGY is not specified.
LSF_HPC_NCPU_COND=and|or	This entry in the lsf.conf file defines how any two LSF_HPC_NCPU_* thresholds are combined. The default value is or.
LSF_HPC_NCPU_INCREMENT=increment	This entry in the lsf.conf file defines the upper limit for the number of processors that are changed since the last checking cycle. The default value is 0.
LSF_HPC_NCPU_INCR_CYCLES=icycles	This entry specifies the minimum number of consecutive cycles in which the number of processors changed does not exceed LSF_HPC_NCPU_INCREMENT. LSF checks total usable processors every 2 minutes.
LSF_HPC_NCPU_THRESHOLD=threshold	This entry specifies the percentage of total usable processors in the LSF partition. The default is 80.
LSF_NON_PRIVILEGED_PORTS=Y|y	Some LSF-HPC communication can occur through privileged ports. This setting disables privileged ports usage ensuring that no communication occurs through privileged ports. Disabling privileged ports helps to ensure system security. By default, LSF daemons and clients running under the root account use privileged ports to communicate with each other. If LSF_NON_PRIVILEGED_PORTS is undefined, and if LSF_AUTH is not defined in lsf.conf file, LSF daemons check the privileged port of request message to perform authentication. If LSF_NON_PRIVILEGED_PORTS=Y is defined, LSF clients (LSF commands and daemons) do not use privileged ports to communicate with daemons, and LSF daemons do not check privileged ports of incoming requests to do authentication. This setting is undefined by default.
LSF_SLURM_DISABLE_CLEANUP=y|Y	This setting disables cleanup of jobs other than LSF jobs running in a SLURM LSF partition. By default, only LSF jobs are allowed to run within a SLURM LSF partition. LSF periodically cleans up any jobs submitted outside of LSF. This cleanup period is defined through LSB_RLA_UPDATE. For example, the following srun job is not submitted through LSF, so it is terminated: % srun -n 4 -p lsf sleep 100000 srun: error: n13: task[0-1]: Terminated srun: Terminating job If LSF_SLURM_DISABLE_CLEANUP=Y is set, this job would be allowed to run. This setting is undefined by default.
LSF_SLURM_TMPDIR=path	This setting specifies the LSF tmp directory for HP XC systems. The default LSF_TMPDIR /tmp cannot be shared across nodes, so LSF_SLURM_TMPDIR must specify a path that is accessible on all nodes of the HP XC system. LSF_SLURM_TMPDIR affects only HP XC configuration. It is ignored on other systems in a mixed cluster environment. The location of LSF tmp directory is determined in the following order: LSF_SLURM_TMPDIR, if defined LSF_TMPDIR, if defined The default shared directory /hptc_cluster/lsf/tmp The default path is /hptc_cluster/lsf/tmp.

Table 13-4 describes the environment variables in the lsb.queues file that you can use to enhance LSF-HPC.

Table 13-4  Environment Variables for LSF-HPC Enhancement (lsb.queues File)

Environment Variable	Description
DEFAULT_EXTSCHED= SLURM[options[;options]...]	This entry specifies SLURM allocation options for the queue. The -ext options to the bsub command are merged with DEFAULT_EXTSCHED options, and -ext options override any conflicting queue-level options set by DEFAULT_EXTSCHED. For example, if DEFAULT_EXTSCHED=SLURM[nodes=2;tmp=100] and a job is submitted with -ext "SLURM[nodes=3;tmp=]", LSF-HPC uses the following resulting options for scheduling: SLURM[nodes=3] The nodes=3 specification in the -ext option overrides the nodes=2 setting in DEFAULT_EXTSCHED, and the tmp= assignment in the -ext option overrides the tmp=100 setting in DEFAULT_EXTSCHED. You can use the DEFAULT_EXTSCHED environment variable in combination with MANDATORY_EXTSCHED environment variable in the same queue. For example: -ext "SLURM[nodes=3;tmp=]" DEFAULT_EXTSCHED=SLURM[nodes=2;tmp=100] MANDATORY_EXTSCHED=SLURM[contiguous=yes;tmp=200] LSF-HPC uses the resulting options for scheduling: SLURM[nodes=3;contiguous=yes;tmp=200] The nodes=3 specification in the -ext option overrides the nodes=2 setting in DEFAULT_EXTSCHED, and the tmp= specification in the -ext option overrides the tmp=100 setting in DEFAULT_EXTSCHED. MANDATORY_EXTSCHED adds the contiguous=yes setting, and overrides the tmp= in the -ext option and tmp=100 in DEFAULT_EXTSCHED with tmp=200. If allocation options are set in DEFAULT_EXTSCHED, and you do not want to specify values for these options, use the keyword with no value in the -ext option of the bsub command. For example, if DEFAULT_EXTSCHED=SLURM[nodes= 2], and you do not want to specify any node option at all, use -ext "SLURM[nodes=]". For more information, see bsub(1).
MANDATORY_EXTSCHED= SLURM[options[;options]...]	This specifies mandatory SLURM allocation options for the queue. The -ext options of the bsub command are merged with MANDATORY_EXTSCHED options, and MANDATORY_EXTSCHED options override any conflicting job-level options set by the -ext option. This setting overrides the -ext options of the bsub command. For example, if MANDATORY_EXTSCHED=SLURM[contiguous=yes;tmp=200] and a job is submitted with -ext "SLURM[nodes=3;tmp=100]", LSF uses the following resulting options for scheduling: "SLURM[nodes=3;contiguous=yes;tmp=200]" You can use the MANDATORY_EXTSCHED environment variable in combination with DEFAULT_EXTSCHED in the same queue. For example: -ext "SLURM[nodes=3;tmp=]" DEFAULT_EXTSCHED=SLURM[nodes=2;tmp=100] MANDATORY_EXTSCHED=SLURM[contiguous=yes;tmp=200] LSF-HPC uses the resulting options for scheduling: SLURM[nodes=3;contiguous=yes;tmp=200] The nodes=3 specification in the -ext option overrides the nodes=2 specification in DEFAULT_EXTSCHED, and tmp= in the -ext option overrides the tmp=100 setting in DEFAULT_EXTSCHED. MANDATORY_EXTSCHED adds contiguous=yes, and overrides tmp= in -ext option and tmp=100 in DEFAULT_EXTSCHED with tmp=200. If you want to prevent users from setting allocation options in the -ext option of the bsub command, use the keyword with no value. For example, if the job is submitted with -ext "SLURM[nodes=4]", use MANDATORY_EXTSCHED= RMS[nodes=] to override this setting. For more information, see bsub(1).

When the HP XC system starts, some computer nodes may take a while to boot. If LSF-HPC starts to report the current number of processors before the system stabilizes, the smaller jobs that are already queued are scheduled. It may be better to run a larger job requesting more processors.

Use the following parameters to control when LSF-HPC starts to report total usable processors on HP XC system: