Task 6: Set Up the System Environment

Table 3-4 lists the tasks that are required to set up the system environment. Perform these tasks now before the system is configured so that the information is propagated to the appropriate client nodes during the initial image synchronization.

Some tasks are required; other tasks are optional depending upon how you want the overall system environment to function, as shown in Table 3-4.

Table 3-4 System Environment Setup Tasks

Required Tasks	Optional Tasks
“Put the License Key File in the Correct Location (Required)”	“Install Additional Software over the Network”
“Configure Interconnect Switch Monitoring Line Cards (Required)”	“Create the /hptc_cluster File System ”
“Configure sendmail (Required)”	“Modify Workstation Model Names in the Database”
“Customize the Nagios Environment (Required)”	“Enable Software RAID-0 or RAID-1 on Client Nodes”
“Set the BMC/IPMI Password on HP Integrity Systems (Required)”	“Create Local User Accounts ”
	“Override Default User and Group Account IDs”
	“Customize Client Node Disk Partitioning”
	“Create the HP Modular Cooling System Configuration File”

When you have finished setting up the system environment, proceed to “Task 7: Run the cluster_config Utility to Configure the System” to begin the system configuration process.

Put the License Key File in the Correct Location (Required)

The HP XC license key file was e-mailed to you, and “Task 7: Have the License Key File Ready” instructed you to save the license key information into a file named XC.lic on another machine. If you have not received the license key file, contact your HP representative.

Follow this procedure to put the license key file in the correct location:

Begin this procedure as the root user on the head node.
Use the file copy utility of your choice (such as scp) to copy the XC.lic file to the /opt/hptc/etc/license directory on the head node.

Make sure the file permissions are set to allow only the user root to have read and write access:

# chmod 600 /opt/hptc/etc/license/XC.lic
# ls -l /opt/hptc/etc/license/XC.lic
-rw-------    1 root     root      941 Oct 20 10:34 XC.lic

The discovery process looks for and reads the XC.lic file in the /opt/hptc/etc/license directory, and there is nothing else you have to do to install an HP XC license.

Configure Interconnect Switch Monitoring Line Cards (Required)

You must configure Quadrics switch controller cards, InfiniBand switch controller cards, and Myrinet monitoring line cards on the interconnect to diagnose and debug problems with the interconnect. These cards must be configured to pass the operation verification program (OVP), which is used to verify the proper operation of the system after it has been installed and configured.

Proceed to one of the following sections depending upon the type of interconnect in use; return to this chapter when you are finished configuring the cards.

Configure sendmail (Required)

LSF requires a mail program to send job output to users who submit jobs and to send administrative messages to the LSF administrator.

By default, LSF uses the sendmail mail program. The sendmail service is installed by default on the head node in /usr/lib/sendmail, but you must configure sendmail to meet your site requirements.

sendmail Configuration Requirements on an HP XC System

Although Linux sendmail typically functions correctly as shipped, current XC host naming conventions cause sendmail to improperly identify itself to other mail servers. This improper identification can lead to the mail being rejected by the remote server.

To remedy this issue, perform the following procedure on all nodes with an external connection that will send mail:

Be sure you know the fully qualified host name associated with the node's external IP address.
On the head node, add an entry for the IP address and fully qualified host name to the top of the /etc/hosts file. Make the entry above the comment #XC-CLUSTER Do Not Edit Below this Line. An entry is similar to the following:
192.0.2.3 penguin.southpole.com
In the previous example, penguin.southpole.com is the fully qualified host name, and 192.0.2.3 is its external IP address.
Save your change and exit the file.
Use the text editor of your choice to open and edit the following file:
/etc/mail/submit.cf

Locate the section of the file that is similar to this:

# my official domain name
# ... define this only if sendmail cannot automatically determine your domain 
#Dj$w.Foo.COM

Uncomment the macro and include the fully qualified host name. The entry is similar to this for penguin.southpole.com:
Djpenguin.southpole.com
Save your change and exit the file.
Use the text editor of your choice to open and edit the following file:
/etc/mail/sendmail.cf
Repeat steps 5 and 6 to make the same changes to this file.
Save your change and exit the file.
Restart sendmail:
# service sendmail restart

To forward mail to users, the sendmail service requires users to create .forward files in their home directories to specify where mail is to be sent.

If you intend to make additional, more advanced modifications to sendmail, HP recommends that you do not modify the .cf files directly. Rather, modify the .mc files and run them through an M4 processor to create the .cf files. The M4 processing tools are part of the sendmail-cf RPM, which is not installed by default. The sendmail-cf RPM is available on the HP XC distribution DVD in the extra RPMs directory.

Consult any third-party Linux system administration reference manual if you need more information about configuring sendmail.

To use a mail program other than sendmail, set the LSB_MAILPROG variable in the /opt/hptc/lsf/top/conf/lsf.conf file to the appropriate mail program after the system configuration is complete.

Customize the Nagios Environment (Required)

Nagios is a highly customizeable system monitoring tool that you can tailor to specific installation and monitoring requirements. HP recommends that you consider certain aspects of the Nagios environment as part of the initial system setup to optimize the type of system events reported to you as well as the frequency of alerts.

Consider the following items when customizing Nagios to suit your environment:

Setting thresholds for load averages
Setting up e-mail contacts
Modifying alert rules files
Setting up event logging
Using event handler scripts to remove suspect nodes from the SLURMlsf partition

Because customizing and tuning the Nagios environment can be done at any time and as often as required, the procedures are documented in the HP XC System Software Administration Guide. See that document and return here when you have completed tuning the Nagios environment.

Set the BMC/IPMI Password on HP Integrity Systems (Required)

Setting the BMC/IPMI password is required on HP Integrity systems. This procedure is documented in the HP XC System Software Administration Guide. Return to this document when you have completed the task.

Install Additional Software over the Network

Perform this task only if you need to install HP software products, third-party software products, or other software over the network.

Because you have run the cluster_prep utility, network connectivity is established on the head node. If you need to install HP software products, third-party software products, or other software over the network (either from a vendor Web site or other remote server) do so now so that the software is transparently propagated to all nodes during the initial image synchronization.

Installation instructions for software products not contained on the HP XC System Software DVD are the responsibility of the software provider and are not provided in this document.

HP StorageWorks Scalable File Share

The HP XC System Software enables Lustre^[1] client services for high-performance and high-availability file I/O. These Lustre client services require the separate installation of Lustre client software, which is provided by the HP StorageWorks Scalable File Share (SFS) product. You can configure HP SFS to operate with an HP XC system to provide scalable storage that is easy to use and easy to administer.

In addition to providing scalable storage, locating the clusterwide /hptc_cluster file system on an HP SFS server enables fail over of that file system. If you want to enable improved availability of this file system, install the HP SFS client software now so that you can mount the file system on the SFS server instead of installing it on a local disk.

The HP SFS client software is distributed on the HP StorageWorks Scalable File Share Client Software CD (the SFS software is not provided on the HP XC System Software DVD). Contact your HP representative if you do not have a copy of the HP SFS CD.

Instructions for installing HP SFS client RPMs and configuration instructions specific to HP XC are located in the SFS documentation, specifically the SFS Client Installation and User Guide. Follow those installation instructions to install SFS now.

The SFS documentation is available on the documentation CD that was delivered with the SFS product. It is also available at the following Web site:

http://www.docs.hp.com/en/highperfcomp.html

Create the /hptc_cluster File System

Do one of the following only if you chose to install the /hptc_cluster file system somewhere other than on the installation disk on the head node.

Mount the /hptc_cluster File System on an HP SFS Server

During the HP XC system configuration process, you might have decided to install the /hptc_cluster file system on an HP SFS server to provide failover capabilities. Thus, if you want this file system located on an HP SFS server, you must mount the mount the /hptc_cluster now. The SFS Client Installation and User Guide describes how to mount the /hptc_cluster file system on an HP SFS file system.

Create and Mount the /hptc_cluster File System Locally

If you intend to put the /hptc_cluster file system on a local disk, you must create and mount the /hptc_cluster file system now on a local disk and partition of your choice.

Remember to reformat the file system using the appropriate utility (such as mkfs) to clear out old data.

Modify Workstation Model Names in the Database

Perform this task if the hardware configuration contains HP wx8400 workstations. Bypass this task if the hardware configuration does not contain this model of HP workstation.

Follow this procedure to correctly identify HP xw8400 workstations in the configuration and management database:

Begin this procedure as the root user on the head node.
Change to the following directory:
# cd /opt/hptc/config/sbin
Modify the workstation model name in the database. In the following command, node_list represents the list of nodes that are xw8400 workstations.
Enter the node_list in the format, node_prefix[n-n], where n is the node number. For example, n[1-3,7,8].
# ./modify_node_type "hp workstation xw8200" \ "HP xw8400 Workstation" node_list

Enable Software RAID-0 or RAID-1 on Client Nodes

This task is optional. Perform this task only if you want to configure software RAID-0 or RAID-1 on client nodes.

You have the option to enable software RAID-0 (striping) or software RAID-1 (mirroring) on client nodes. RAID is an acronym for redundant array of inexpensive (or independent) disks. RAID is a way of combining multiple disks into a single entity to improve performance or reliability or both.

Software RAID-0 (striping) enables client nodes that have more than one storage disk to split data evenly across the disks. Striping is typically used to increase performance. However, because there is no parity information for redundancy, reliability is reduced. A single disk failure affects both disks. Software RAID-0 is not supported on the head node.

When a client node is imaged with software RAID-0 support, the /boot partition is mirrored and swap partitions on each disk are not striped or mirrored. Linux automatically stripes across swap partitions when the swap partitions are on separate disks.

Software RAID-1 enables client nodes that have more than one storage disk to increase reliability by maintaining a redundant (mirrored) disk copy. When a node is imaged with RAID-1 support, the entire disk is mirrored. If a disk fails, nodes can continue to run using the other disk.

To enable software RAID-1 on a client node, the following criteria apply:

The client node must have two or more disks; only two disks are used in the RAID set.
Disks must be the same size.

Follow this procedure to configure software RAID-0 or RAID-1 on one or more client nodes:

Use the text editor of your choice to edit the following configuration file on the head node:
/etc/systemimager/systemimager.conf
Add the following line or lines to the bottom of the file to enable software RAID-0 or RAID-1. Replace node_prefix[n-n] with the node prefix and the range of nodes on which you want to enable software RAID (for example n[1-4,7,9]). You must include a space before and after the equal sign (=).
SOFTWARE_RAID0_NODES = node_prefix[n-n] SOFTWARE_RAID1_NODES = node_prefix[n-n]
Save your change and exit the file.

Create Local User Accounts

This task is optional. If you intend to create local user accounts on the HP XC system rather than manage user accounts through another user authentication method (such as NIS or LDAP), use the Linux adduser command to create local user accounts on the system now, before the system is configured.

See the HP XC System Software Administration Guide if you need more information about creating local user accounts.

Special Considerations for Nagios and LSF

During the system configuration phase, the cluster_config command attempts to create a nagios and an lsfadmin account for use by Nagios and LSF, respectively.

To use existing nagios and lsfadmin user accounts from a site wide NIS system (or some other external user authentication system), you must manually create local XC accounts that mirror the site wide accounts (with matching user identification (UID) and group identification (GID) values). In that way, the cluster_config command uses these local accounts to properly configure Nagios and LSF with the right file permissions, access, and so on.

Override Default User and Group Account IDs

This task is optional. Perform this task only if the default user and group IDs are not suitable for your environment.

Table 3-5 lists the user and group account IDs that are configured by default on an HP XC system if they are not already in use. If any of the default user and group identifiers conflict with other accounts or are not suitable for your environment, you can override them by creating the user accounts manually now (before running the cluster_config utility).

Table 3-5 HP XC Default User and Group Account IDs

User Name	UID	Group Name	GID
`nagios`	`222`	`hpasm`	`222`
`qsnet`	`232`	`qsnet`	`232`
`slurm`	`500`	`slurm`	`500`
`lsfadmin`	`501`	`lsfadmin`	`501`
`hpsmh`	`79`	`hpsmh`	`79`

Customize Client Node Disk Partitioning

This task is optional. Perform this task only if you want to change the default partition layout on client nodes.

The HP XC client node imaging process requires a single system disk on each client node for the operating system installation. Client node disk devices are partitioned automatically as part of the automated client imaging process.

Table 3-6 lists the default disk partition layout on client nodes. Partition sizes are created as a percentage of available disk space after subtracting the total swap size.

Table 3-6 Default Client Node Partition Layout

File System Name	Size
One swap partition	Swap space is calculated based on the amount of memory on the node, and it is governed by minimum and maximum values you set (see Appendix E for more information). By default, swap space is calculated as 100% x (total memory).
`/boot` or `/boot/efi`	1% x (total disk size minus swap space)
`/` (root)	60% x (total disk size minus swap space)
`/var`	39% x (total disk size minus swap space)

If the default client node disk partition layout is satisfactory for your needs, you do not have to do anything. However, if the system requires a different, customized partitioning layout on client disk devices, see Appendix E for instructions.

Create the HP Modular Cooling System Configuration File

This task is optional. Perform this task only if the hardware configuration includes a water-based HP Modular Cooling System (MCS) device.

Follow this procedure to create and populate an MCS configuration file called mcs.ini. The discover command uses the information in this file to discover the MCS devices.

Begin this procedure as the root user on the head node.
Change to the following directory:
# cd /opt/hptc/mcs/config
Copy the MCS template file into a file called mcs.ini:
# cp mcs_template.ini mcs.ini
Use the text editor of your choice to populate the mcs.ini file with site-specific MCS device information. A sample of the mcs.ini file is shown in Example 3-1.
The following list describes the parameters in the mcs.ini file:
mcs_units
Specifies each MCS device name separated by a comma.
mcs_server_units
Specifies the name of each MCS device that controls one or more service nodes, separated by a comma
name
Specifies the name of the MCS device.
ipaddr
Specifies the IP address of the MCS device.
location
Specifies the cabinet that contains the nodes that are under control of the MCS device
nodes
Defines the range of nodes under control of the MCS device. The node range is specified in the format node_prefix[n-n].
status
Lists the status of the MCS device. Valid values are offline or online.
Nagios monitors only the MCS devices that are in the online state. For more information about MCS device status, see manage_mcs_status(8).
Before you change the device status to online, make sure the MCS devices are online and operating correctly.
Save your changes and exit the file.

Example 3-1 Sample mcs.ini File

# mcs.ini file for cluster penguin

[global]
mcs_units=mcs1,mcs2,mcs3,mcs4,mcs5
mcs_server_units=mcs5

[mcs1]
name=mcs1
ipaddr=172.23.0.1
location=Cab CBB1
nodes=n[1-36]
status=offline

[mcs2]
name=mcs2
ipaddr=172.23.0.2
location=Cab CBB2
nodes=n[37-72]
status=offline

[mcs3]
name=mcs3
ipaddr=172.23.0.3
location=Cab CBB3
nodes=n[73-108]
status=offline

[mcs4]
name=mcs4
ipaddr=172.23.0.4
location=Cab CBB4
nodes=n[109-144]
status=offline

[mcs5]
name=mcs5
ipaddr=172.23.0.5
location=Cab CBB5
nodes=n[145-180]
status=offline

^[1]Lustre® technology developed by Cluster File Systems, Inc.