Task 6: Set Up the System Environment

Put the License Key File in the Correct Location (Required)

The HP XC license key file was e-mailed to you, and “Task 7: Have the License Key File Ready” instructed you to save the license key information into a file named XC.lic on another machine. If you have not received the license key file, contact your HP representative.

Follow this procedure to put the license key file in the correct location:

The discovery process looks for and reads the XC.lic file in the /opt/hptc/etc/license directory, and there is nothing else you have to do to install an HP XC license.

Configure Interconnect Switch Monitoring Line Cards (Required)

You must configure Quadrics switch controller cards, InfiniBand switch controller cards, and Myrinet monitoring line cards on the interconnect to diagnose and debug problems with the interconnect. These cards must be configured to pass the operation verification program (OVP), which is used to verify the proper operation of the system after it has been installed and configured.

Proceed to one of the following sections depending upon the type of interconnect in use; return to this chapter when you are finished configuring the cards.

Configure sendmail (Required)

LSF requires a mail program to send job output to users who submit jobs and to send administrative messages to the LSF administrator.

By default, LSF uses the sendmail mail program. The sendmail service is installed by default on the head node in /usr/lib/sendmail, but you must configure sendmail to meet your site requirements.

sendmail Configuration Requirements on an HP XC System

Although Linux sendmail typically functions correctly as shipped, current HP XC host naming conventions cause sendmail to improperly identify itself to other mail servers. This improper identification can lead to the mail being rejected by the remote server.

To remedy this issue, perform the following procedure on all nodes with an external connection that will send mail:

To forward mail to users, the sendmail service requires users to create .forward files in their home directories to specify where mail is to be sent.

If you intend to make additional, more advanced modifications to sendmail, HP recommends that you do not modify the .cf files directly. Rather, modify the .mc files and run them through an M4 processor to create the .cf files. The M4 processing tools are part of the sendmail-cf RPM, which is not installed by default. The sendmail-cf RPM is available on the HP XC distribution DVD in the extra RPMs directory.

Consult any third-party Linux system administration reference manual if you need more information about configuring sendmail.

To use a mail program other than sendmail, set the LSB_MAILPROG variable in the /opt/hptc/lsf/top/conf/lsf.conf file to the appropriate mail program after the system configuration is complete.

Customize the Nagios Environment (Required)

Nagios is a highly customizeable system monitoring tool that you can tailor to specific installation and monitoring requirements. HP recommends that you consider certain aspects of the Nagios environment as part of the initial system setup to optimize the type of system events reported to you as well as the frequency of alerts.

Consider the following items when customizing Nagios to suit your environment:

Because customizing and tuning the Nagios environment can be done at any time and as often as required, the procedures are documented in the HP XC System Software Administration Guide. See that document and return here when you have completed tuning the Nagios environment.

Configure Access to the Console Port on the Head Node (Required)

Perform this task only if the console port on the head node is connected to the external network. Otherwise, omit this task.

To monitor sensor data and system event logs, Nagios requires that the console port on the head node be accessible. The procedure is documented in the HP XC System Software Administration Guide. Return here when you have completed the task.

Set the BMC Password on HP Integrity Systems (Required)

To monitor sensor data and system event logs, Nagios requires that you set a base motherboard controller (BMC) password on HP Integrity systems. This procedure is documented in the HP XC System Software Administration Guide. Return here when you have completed the task.

Install Additional Software From HP or Third-Party Vendors

This task is optional. Perform this task only if you need to install additional HP software products, third-party software products, or other software that is required for your computing environment. Otherwise, omit this task.

Install the software now before the system is configured so that the software is transparently propagated to all nodes during the initial image synchronization.

The remainder of this section provides information about installing additional software products on an HP XC system. The following topics are addressed:

Install Additional HP Software Products

This section lists prominent HP software products that you can integrate into an HP XC system. You might require some of these products depending upon how you plan to configure the HP XC system software environment. You must purchase and license these products from HP.

This section does not contain product-specific installation information for other HP software products; see the product documentation for product-specific installation instructions.

• “HP StorageWorks Scalable File Share”

• “HP Serviceguard”

• “HP Remote Graphics Software”

	NOTE: HP SFS, Serviceguard, and RGS have not been qualified on HP XC System Software that is installed on Red Hat Enterprise Linux (described in Chapter 7). If you are installing HP XC System Software on Red Hat Enterprise Linux, do not install these HP software products.

HP StorageWorks Scalable File Share

The HP XC System Software enables Lustre^[1] client services for high-performance and high-availability file I/O. These Lustre client services require the separate installation of Lustre client software, which is provided by the HP StorageWorks Scalable File Share (SFS) product. You can configure HP SFS to operate with an HP XC system to provide scalable storage that is easy to use and easy to administer.

In addition to providing scalable storage, locating the clusterwide /hptc_cluster file system on an HP SFS server enables fail over of that file system. If you want to enable improved availability of this file system, install the HP SFS client software now so that you can mount the file system on the SFS server instead of installing it on a local disk.

The HP SFS client software is distributed on the HP StorageWorks Scalable File Share Client Software CD (the SFS software is not provided on the HP XC System Software DVD). Contact your HP representative if you do not have a copy of the HP SFS CD.

Instructions for installing HP SFS client RPMs and configuration instructions specific to HP XC are located in the SFS documentation, specifically the SFS Client Installation and User Guide. Follow those installation instructions to install SFS now.

The SFS documentation is available on the documentation CD that was delivered with the SFS product. It is also available at the following website:

http://www.docs.hp.com/en/linuxhpc.html

HP Serviceguard

HP Serviceguard is the service availability tool recommended for use on an HP XC system.

If you intend to use Serviceguard as the availability tool, you must have in your possession the Serviceguard license, the distribution media, and the product documentation. Use the Serviceguard documentation to install the product now. The documentation is available from the following website:

http://www.docs.hp.com/en/ha.html

The HP XC System Software has been tested with the following Serviceguard for Linux RPMs. You must install the RPMs in the following order:

1. pidentd-3.0.15sg-1.rpm

2. qs-A.02.00.03-0.product.redhat.{architecture}.rpm

3. serviceguard-A.11.16.04-0.product.redhat.{architecture}.rpm

	NOTE: HP recommends that you obtain the latest available release of Serviceguard for Linux Version 11.16 and all available patches so that you install the most up-to-date version of Serviceguard. New versions of Serviceguard Version 11.16 interoperate with HP XC Version 3.2 as expected.

The quorum server RPM, qs-A.02.00.03-0.product.redhat.{architecture}, is required only if you plan to configure a quorum server. If your particular configuration uses lock LUNs exclusively, you do not install the quorum server RPM.

The Serviceguard software contains two modules that are built against the kernel.

After installing the Serviceguard software, proceed to “Deciding on the Method to Achieve Quorum for HP Serviceguard Clusters” to define and configure the method to achieve quorum.

Deciding on the Method to Achieve Quorum for HP Serviceguard Clusters. In a Serviceguard configuration, each availability set becomes its own two-node Serviceguard cluster, and each Serviceguard cluster requires some form of quorum. The quorum acts as a tie breaker in the Serviceguard cluster running on each availability set. If connectivity is lost between the nodes of the Serviceguard cluster, the node that can access the quorum continues to run the cluster and the other node is considered down.

The quorum can be either a quorum server or a lock LUN; you must configure one or the other on every availability set. You can configure a lock LUN as the tie breaker only if the head node and the other node in the availability set are both connected to the same shared storage (for instance, an MSA) and both are able to access the same partition.

Configuring a Quorum Server. To configure a quorum server, all you have to do is to supply the node name of the quorum server when prompted by the cluster_config utility; there is nothing you need to do now.

You can select any node in the HP XC system that is not participating in any availability set to serve as the quorum server, even a compute node. You can use the same quorum server for one or more availability sets. If you configure a quorum server, you must have previously installed the qs-A.02.00.03-0.product.redhat.{architecture}.rpm RPM.

Configuring a Lock LUN. If you intend to use a lock LUN instead of a quorum server to achieve quorum, enter the following command to create the lock LUN now, before running the cluster_config utility later in the system configuration process.

In the following command, /dev/sdb is the full path to the disk on the MSA, and partition 1 on that disk is configured as the lock LUN:

# /sbin/fdisk /dev/sdb Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 Command (m for help): t Selected partition 1 HEX code (type L to list codes): 83 Command (m for help): w

When you run the cluster_config utility, it prompts you to supply the name of the lock LUN, and you must supply the full path with partition (for example, /dev/sdb1, where 1 is the partition number).

HP Remote Graphics Software

HP Remote Graphics Software (RGS) is an optional software product that displays images created on a remote SVA display device. If you want to use RGS with SVA, you must purchase RGS separately from HP.

See the RGS product documentation for installation instructions. If you install RGS, the HP XC cluster_config utility prompts you for specific RGS configuration information when you configure the system.

If you install RGS and intend to use it from the head node, follow this procedure to modify the following configuration file:

1. Use the text editor of your choice to open the /etc/X11/xorg.conf file.

2. Add the following line to the section labeled “Module”:

   Load “rge”

3. Save your changes to the file and exit the text editor.

Create the /hptc_cluster File System

This task is optional. Do one of the following only if you chose to install the /hptc_cluster file system somewhere other than on the installation disk on the head node.

Mount the /hptc_cluster File System on an HP SFS Server

During the HP XC system configuration process, you might have decided to configure the /hptc_cluster file system on an HP SFS server to provide failover capabilities.

Thus, if you want this file system located on an HP SFS server, you installed the HP SFS software as described in “HP StorageWorks Scalable File Share”, and you must mount /hptc_cluster now. The SFS Client Installation and User Guide describes how to mount the /hptc_cluster file system on an HP SFS file system.

Create and Mount the /hptc_cluster File System Locally

If you intend to put the /hptc_cluster file system on a local disk, you must create and mount the /hptc_cluster file system now on a local disk and partition of your choice.

Remember to reformat the file system using the appropriate utility (such as mkfs) to clear out old data.

Modify Workstation Model Names in the Database

This task is optional. Perform this task if the hardware configuration contains HP xw8400 or xw9400 workstations. Omit this task if the hardware configuration does not contain these models of HP workstations.

Follow this procedure to correctly identify HP xw8400 or xw9400 workstations in the configuration and management database:

Enable Software RAID-0 or RAID-1 on Client Nodes

This task is optional. Perform this task only if you want to configure software RAID-0 or RAID-1 on client nodes.

You have the option to enable software RAID-0 (striping) or software RAID-1 (mirroring) on client nodes. RAID is an acronym for redundant array of inexpensive (or independent) disks. RAID is a way of combining multiple disks into a single entity to improve performance or reliability or both.

Software RAID-0 (striping) enables client nodes that have more than one storage disk to split data evenly across the disks. Striping is typically used to increase performance. However, because there is no parity information for redundancy, reliability is reduced. A single disk failure affects both disks. Software RAID-0 is not supported on the head node.

When a client node is imaged with software RAID-0 support, the /boot partition is mirrored and swap partitions on each disk are not striped or mirrored. Linux automatically stripes across swap partitions when the swap partitions are on separate disks.

Software RAID-1 enables client nodes that have more than one storage disk to increase reliability by maintaining a redundant (mirrored) disk copy. When a node is imaged with RAID-1 support, the entire disk is mirrored. If a disk fails, nodes can continue to run using the other disk.

To enable software RAID-1 on a client node, the following criteria apply:

Follow this procedure to configure software RAID-0 or RAID-1 on one or more client nodes:

Create Local User Accounts

This task is optional. If you intend to create local user accounts on the HP XC system rather than manage user accounts through another user authentication method (such as NIS or LDAP), use the Linux adduser command to create local user accounts on the system now, before the system is configured.

See the HP XC System Software Administration Guide if you need more information about creating local user accounts.

Special Considerations for Nagios and LSF

During the system configuration phase, the cluster_config command attempts to create a nagios and an lsfadmin account for use by Nagios and LSF, respectively.

To use existing nagios and lsfadmin user accounts from a site wide NIS system (or some other external user authentication system), you must manually create local XC accounts that mirror the site wide accounts (with matching user identification (UID) and group identification (GID) values). In that way, the cluster_config command uses these local accounts to properly configure Nagios and LSF with the right file permissions, access, and so on.

Override Default User and Group Account IDs

This task is optional. Perform this task only if the default user and group IDs are not suitable for your environment.

Table 3-5 lists the user and group account IDs that are configured by default on an HP XC system if they are not already in use. If any of the default user and group identifiers conflict with other accounts or are not suitable for your environment, you can override them by creating the user accounts manually now (before running the cluster_config utility).

Customize Client Node Disk Partitioning

This task is optional. Perform this task only if you want to change the default partition layout on client nodes.

The HP XC client node imaging process requires a single system disk on each client node for the operating system installation. Client node disk devices are partitioned automatically as part of the automated client imaging process.

Table 3-6 lists the default disk partition layout on client nodes. Partition sizes are created as a percentage of available disk space after subtracting the total swap size.

If the default client node disk partition layout is satisfactory for your needs, you do not have to do anything.

However, if the client nodes require a different partitioning scheme, you have the flexibility to apply customized, fixed partition sizes or to assign partition sizes based on a percentage of the disk size to all or a subset of client nodes.

See Appendix E for instructions.

Create the HP Modular Cooling System Configuration File

This task is optional. Perform this task only if the hardware configuration includes a water-based HP Modular Cooling System (MCS) device.

Follow this procedure to create and populate an MCS configuration file called mcs.ini. The discover command uses the information in this file to discover the MCS devices.

Example 3-3 Sample mcs.ini File

# mcs.ini file for cluster penguin [global] mcs_units=mcs1,mcs2,mcs3,mcs4,mcs5 mcs_server_units=mcs5 [mcs1] name=mcs1 ipaddr=172.23.0.1 location=Cab CBB1 nodes=n[1-36] status=offline [mcs2] name=mcs2 ipaddr=172.23.0.2 location=Cab CBB2 nodes=n[37-72] status=offline [mcs3] name=mcs3 ipaddr=172.23.0.3 location=Cab CBB3 nodes=n[73-108] status=offline [mcs4] name=mcs4 ipaddr=172.23.0.4 location=Cab CBB4 nodes=n[109-144] status=offline [mcs5] name=mcs5 ipaddr=172.23.0.5 location=Cab CBB5 nodes=n[145-180] status=offline

Mount Network File Systems

This task is optional. If you plan to mount NFS file systems, add the mount points to the /hptc_cluster/etc/fstab.proto file now so that the mount points are propagated to the golden image.

	NOTE: See the HP XC System Software Administration Guide if you need more information about how to modify the /hptc_cluster/etc/fstab.proto file.

Update initrd Files With Required Hardware

This task is optional. When client nodes contain additional hardware that is required for booting (such as SCSI controllers) and the hardware is not present on the head node, you must modify the initrd file on the head node to include the drivers.

	NOTE: Perform this task now before running the cluster_config utility. Performing this task after running the cluster_config utility requires you to remove the golden image.

Example 3-4 provides a sample script that updates all initrd files if a client node contains the Adaptec SCSI controller:

Example 3-4 Script To Update initrd Files With Hardware Required For Booting

#!/bin/bash FILES=`find /boot -name initrd\* -print` for i in $FILES do VERSION=`echo $i | sed -e 's/^.*initrd-\(.*\).img/\1/'` if [ ! -z "$VERSION" ] then NEWF=/tmp/initrd-$VERSION.img echo "/sbin/mkinitrd --with=aacraid -f $NEWF $VERSION" /sbin/mkinitrd --with=aacraid -f $NEWF $VERSION if [ -f $NEWF ] then echo "replacing $i with $NEWF" rm $i mv $NEWF $i fi fi done