LSF-HPC Troubleshooting

Take the following steps if you are have trouble submitting jobs or controlling LSF-HPC:

Ensure that the number of nodes in the lsf partition is less than or equal to the number of nodes reported in the XC.lic file. Sample entries follow:
INCREMENT XC-CPUS Compaq auth.number exp. date nodes ...
INCREMENT XC-PROCESSORS Compaq auth.number exp. date nodes ...
The value for nodes in the XC-CPUS or XC-PROCESSORS entry specifies the number of licensed nodes for this system. If this value does not match the number of actual nodes, the LSF service may fail to start LSF.
Use the lshosts command to determine the number of processors reported by LSF.
Ensure that the date is synchronized throughout the HP XC system.
Verify that the /hptc_cluster directory (file system) is properly mounted on all nodes. SLURM relies on this file system.
Ensure that SLURM is configured, up, and running properly.
Examine the SLURM log files in /var/slurm/log/ directory on the SLURM master node for any problems.
If the sinfo command reports that the node is down and daemons are running, examine the available processors vs. Procs setting in the slurm.conf file.
Ensure that the lsf partition is configured correctly.
Verify that the system licensing is operational. Use the lmstat -a command.
Ensure that munge is running on all compute nodes.
If you are experiencing LSF communication problems, examine for potential firewall issues.
When LSF-HPC failover is disabled and the LSF execution host (which is not the head node) goes down, issue the controllsf command to restart LSF-HPC on the HP XC system:
# controllsf start
When failover is enabled, you need to intervene only when the primary LSF execution host is not started on HP XC system startup (when the startsys command is run). Use the controllsf command to restart LSF-HPC.
# controllsf start
When starting LSF-HPC after a partial system shutdown, LSF is started on the head node if:
- LSF failover is enabled.
- The head node has the "resource management" role and no other resource management node is up.
- The head node has the “resource management” role and the enable headnode preferred subcommand is set.
- LSF-HPC was not shut down cleanly, perhaps as a result of running startsys without running service lsf stop or controllsf stop on the head node.
- LSF-HPC starts on the head node if the other resource management nodes are unavailable
LSF-HPC failover may select the node that it just released.
LSF-HPC failover attempts to ensure that a different node is used after it removes control from the present node. However, if all other options are exhausted, LSF-HPC failover tries the current node again before giving up.
If you are trying to perform load balancing, log in to the primary LSF execution host node and execute the controllsf start here command from that node.

LSF-HPC Troubleshooting

Technical documentation

» Table of Contents

» Glossary

» Index