LSF-HPC with SLURM Monitoring

LSF-HPC with SLURM is monitored and controlled by Nagios using the check_lsf plug-in.

When LSF-HPC with SLURM is down, the response of the check_lsf plug-in depends on whether LSF-HPC with SLURM failover is enabled or disabled:

When LSF-HPC with SLURM failover is disabled
The check_lsf plug-in returns an immediate failure notification to Nagios.
When LSF-HPC with SLURM failover is enabled
The check_lsf plug-in decides if LSF-HPC with SLURM is supposed to be running. If so, it acquires a list of resource management nodes and tries to restart LSF-HPC with SLURM on each of those nodes, in turn, until one succeeds, or until the list is exhausted.
If successful, the check_lsf plug-in returns an LSF OK - restarted message.
If the restart procedure fails, the check_lsf plug-in returns a failure notification.

If the node hosting LSF-HPC with SLURM becomes unresponsive, the Nagios check_lsf plug-in takes action.

Table 15-2 lists the Nagios messages for LSF failover monitor status:

Table 15-2 Nagios Messages for LSF-HPC with SLURM Failover Monitor Status

Message	Meaning
`LSF OK - up`	The LSF-HPC with SLURM environment appears to be up and operational on the HP XC system
`LSF OK - currently shut down`	The LSF-HPC with SLURM environment has not been started on the HP XC system
`LSF CRITICAL - down`	LSF-HPC with SLURM is not running, and LSF-HPC with SLURM failover is disabled
`LSF warning - restarted`	The LSF-HPC with SLURM environment was not running, and should have been; it is being restarted. The message changes to `LSF OK - up` the next time Nagios is updated.
`LSF CRITICAL - {message}`	An abnormal problem occurred. The {`message`} text provides useful diagnostic information.

Technical documentation