Task 18: Finalize the Configuration of Compute Resources

Begin this procedure as the root user on the head node.

Use the SLURM postconfiguration utility to update the slurm.conf file with the correct processor count and memory size. If a QsNet^II interconnect switch is present, an elanhosts file is created with an entry for each compute node. The elanhosts file is part of the QsNet^II switch support that is provided by SLURM.

# spconfig
Detected the Quadrics Elan interconnect; configuring SLURM support...
  The new 'elanhosts' file contains entries for 16 nodes...
  Configured nodes n[1-15] with 2 CPUs and 8108 MB of total memory...
  Configured node n16 with 4 CPUs and 9549 MB of total memory...
  Restarting SLURM...
  SLURM Post-Configuration Done.

In this example, the spconfig utility was run on a system with a QsNet^II interconnect. For all other interconnect types, the first two lines of the command output are not displayed.

	NOTE: If a compute node did not boot up, the spconfig utility configures the node as follows: Configured unknown node n14 with 1 CPU and 1 MB of total memory... After the node has been booted up, re-run the spconfig utility to configure the correct settings.

If the system is using a QsNet^II interconnect, ensure that the number of node entries in the /opt/hptc/libelanhosts/etc/elanhosts file matches the expected number of operational nodes in the cluster. If the number does not match, verify the status of the nodes to ensure that they are all up and running, and re-run the spconfig utility. If the number does not match, it is possible that the QsNet^II network card on the missing node is not fully operational.

Follow this procedure to add customizations to the SLURM configuration file:

1. Go to Appendix J to determine the type of customizations that are available or required. For instance, if you installed and configured SVA, SVA requires certain SLURM customizations.

2. Use the text editor of your choice to edit the SLURM configuration file:

   /hptc_cluster/slurm/etc/slurm.conf

3. Use the information in Appendix J to customize the SLURM configuration according to your requirements.

4. If you make changes to the slurm.conf file, save your changes and exit the text editor.

5. Update the SLURM daemons with this new information:

   # scontrol reconfig

Begin this procedure as the root user on the head node.

Set up the LSF environment by sourcing the LSF profile file:

# . /opt/hptc/lsf/top/conf/profile.lsf

Verify that the LSF profile file has been sourced by finding an LSF command:

# which lsid
/opt/hptc/lsf/top/6.2/linux2.6-glibc2.3-x86_64-slurm/bin/lsid

This sample output was obtained from an HP ProLiant server. Thus, the directory name linux2.6-glibc2.3-x86_64-slurm is included in the path (the string x86_64 signifies a Xeon- or Opteron-based architecture). The string ia64 is included in the directory name for HP Integrity servers. The string slurm exists in the path only if LSF-HPC with SLURM is configured.

	Remainder Applies to LSF-HPC with SLURM: The remainder of this procedure applies to LSF-HPC with SLURM. If standard LSF is configured, omit the remaining steps.

If you assigned two or more nodes with the resource_management role and want to enable LSF failover, enter the following command; otherwise, proceed to step 5.

# controllsf enable failover

Determine the node on which the LSF daemons are running:

# controllsf show current
LSF is currently running on node n32, and assigned to node n32

Log in to the node that is running LSF if it is not running on the head node. If LSF is running on the head node, omit this step.

# ssh n32

Restart the LIM daemon:

# lsadmin limrestart
Checking configuration files ...
   No errors found.
   
   Restart LIM on <lsfhost.localdomain> ...... done

Restarting the LIM daemon is required because the licensing of LSF-HPC with SLURM occurs when the LIM daemon is started. This means that the LIM daemon is licensed only for the processors that are actually available at that time, which might be fewer than the total number of processors available after all of the nodes have been imaged and are up and running.

Update the LSF batch system with the latest resource information reported by SLURM:

# badmin reconfig
Checking configuration files ...

     No errors found.

     Reconfiguration initiated