Task 5: Define and Set Up Your System Environment

Table 4-2 lists the tasks that are required to set up your system environment. Perform these tasks now before the system is configured so that the information is propagated to the appropriate client nodes during the initial image synchronization. Some of the tasks are optional depending upon how you want your overall system environment to function; other tasks are required, as noted.

Table 4-2 System Setup Tasks

Required Tasks	Optional Tasks
Install Patches or RPM Updates (Required)	Create the /hptc_cluster File System (Optional)
Put the License Key File in the Correct Location (Required)	Create Local User Accounts (Optional)
Configure the Interconnect Switch Monitoring Line Cards (Required)	Override Default User and Group Account IDs (Optional)
Configure sendmail (Required)	Customize Client Node Disk Partitioning (Optional)

When you have finished setting up your system environment, proceed to “Task 6: Run the cluster_config Utility to Configure the System” to begin the system configuration process.

Install Patches or RPM Updates (Required)

See the following sources to determine if additional software packages (patches) are available for this release:

The HP IT Resource Center (ITRC) Web site at:
http://www.itrc.hp.com/
“Procedure to Download Patches” describes how to download XC patches from this Web site, if they are available
The HP XC System Software Release Notes
Because the HP XC System Software Release Notes are updated periodically, they are only available on line at:
http://www.hp.com/techservers/clusters/xc_clusters.html

Install the updated RPM packages now before the system is configured so that the software updates are propagated to all nodes during the initial image synchronization.

Procedure to Download Patches

Follow this procedure to download XC patches from the ITRC Web site:

Create a temporary directory for the patch or patches on the head node. You can name this temporary directory anything you want; this procedure creates a directory called /root/patches:
# mkdir /root/patches
Register as an Americas/Asia Pacific or European customer at the following HP remote server:
http://www.itrc.hp.com/
Note:
Because this is a newly-installed system, you must set up the proxies on your Web browser in order to access this Web site.
After supplying information about yourself and your company, an ITRC user ID is assigned to you; you supply your own password. Remember this user ID and password because you must use it whenever you download a patch.
From the registration confirmation window, select the option to go directly to the ITRC home page.
From the IT Resource Center home page, select patch/firmware database from the maintenance and support (hp products) list.
From the patch / firmware database page, select Linux under find individual patches.
From the search for patches page, in step 1 of the search utility, select vendor and version, select hpxc as the vendor and select the XC version that is appropriate for your cluster platform.
In step 2 of the search utility, How would you like to search?, select Browse Patch List.
Click the search>>> button to begin the search.
Download all patches that are available for your cluster platform into the temporary directory you created.
If you are unable to retrieve the files from this Web site, contact the XC support team at:
<xc_support@hp.com>
Follow the patch installation instructions in the README file.

Put the License Key File in the Correct Location (Required)

The HP XC license key file was e-mailed to you, and “Task 6: Have the License Key File Ready” instructed you to save the license key information into a file named XC.lic on another machine. If you have not received the license key file, contact your HP representative.

Follow this procedure to put the license key file in the correct location:

Begin this procedure as the root user on the head node.
Use the file copy utility of your choice (such as scp) to copy the XC.lic file to the /opt/hptc/etc/license directory on the head node.

Make sure the file permissions are set to allow only the user root to have read and write access:

# chmod 600 /opt/hptc/etc/license/XC.lic
# ls -l /opt/hptc/etc/license/XC.lic
-rw-------    1 root     root      941 Dec 20 10:34 XC.lic

During the system configuration procedure, the XC.lic file is discovered in the /opt/hptc/etc/license directory, and there is nothing else you have to do to install an HP XC license.

Configure the Interconnect Switch Monitoring Line Cards (Required)

You must configure the Quadrics switch controller cards, the InfiniBand switch controller cards, and the Myrinet monitoring line cards on the system interconnect in order to diagnose and debug problems with the system interconnect.

Appendix D describes how to configure the line monitoring cards for each interconnect type. Return to this chapter when you are finished configuring the cards.

Configure sendmail (Required)

LSF requires a mail program to send job output to users submitting jobs and to send administrative messages to the LSF administrator.

By default, LSF uses the mail program /usr/lib/sendmail. The sendmail service is installed by default on the head node in /usr/lib/sendmail, but you must configure it to meet your site requirements.

sendmail Configuration Requirements on an HP XC System

Although Linux sendmail typically functions correctly as shipped, current XC host naming conventions cause sendmail to improperly identify itself to other mail servers. This improper identification can lead to the mail being rejected by the remote server.

This problem is easily remedied by taking the following steps on all nodes with an external connection that will be sending mail:

Be sure you know the fully qualified host name associated with the node's external IP address.
On the head node, add an entry for the IP address and fully qualified domain name to the top of the /etc/hosts file. Make your entry above the comment #XC-CLUSTER Do Not Edit Below this Line . Your entry should look similar to the following
192.0.2.3 southpole.com
In the previous example, southpole.com is the fully qualified domain name, and 192.0.2.3 is its external IP address.
Save your change and exit the file.
Use the text editor of your choice to open and edit the following file:
/etc/mail/submit.cf

Locate the section of the file that looks like this:

# my official domain name
# ... define this only if sendmail cannot automatically determine your domain 
#Dj$w.Foo.COM

Uncomment the macro and include your fully qualified host name. The entry should look similar to the following:
Djfully.qualified.name.com
Using our example domain name, the line looks like this:
Djsouthpole.com
Save your change and exit the file.
Restart sendmail:
# service sendmail restart

Users may need to have .forward files in their home directories to specify where mail is to be sent in order for the sendmail service to properly forward mail to users.

Consult any third-party Linux system administration reference manual if you need more information about configuring sendmail.

To use a mail program other than sendmail, you must set the LSB_MAILPROG variable in the /opt/hptc/lsf/top/conf/lsf.conf file to the appropriate mail program after the system configuration is complete.

Create the /hptc_cluster File System (Optional)

During the software installation process, you had the option to create a disk partition for the /hptc_cluster file system somewhere other than the installation disk on the head node.

Do one of the following now if you did not create the /hptc_cluster file system on the system disk on the head node:

Install HP SFS Client RPMs To Configure the /hptc_cluster File System On An HP SFS Server

HP StorageWorks Scalable File Share (HP SFS) is based on Lustre® technology developed by Cluster File Systems, Inc. HP SFS is a turnkey Lustre system that can be configured to operate with an HP XC system.

During the HP XC system configuration process, you may have decided to install the /hptc_cluster file system on an HP SFS server. The procedure documented here describes how to install the SFS client RPMs and mount an SFS file system on your HP XC system to prepare to configure the file system on the SFS server.

Even if you are not installing the /hptc_cluster file system on an HP SFS server, you may already have plans to install the HP SFS client for use with your HP XC system. In that case, install the HP SFS client RPMs now.

The HP SFS client software is distributed on the HP StorageWorks Scalable File Share Client Software CD (it is not provided on the HP XC System Software DVD). Contact your HP representative if you do not have a copy of the HP SFS CD.




	Important: Before you begin this procedure, ensure that the SFS file systems, including `/hptc_cluster` and other user file systems, have been previously created and exported to the HP XC system. The HP StorageWorks Scalable File Share user manuals contain a detailed product overview and describes how to install and configure the servers and storage in the HP SFS system, and how to set up file systems. It also provides information on mounting and unmounting HP SFS file systems on client nodes.

Insert the HP SFS Client Software CD into the drive on the head node.
Mount the CD at the following location:
# mount /dev/cdrom /mnt/cdrom
Change to the following directory :
# cd /mnt/cdrom/architecture/XC_3.0
In the previous path, replace architecture with the appropriate directory name based on system architecture type:
Architecture Type Directory Name
Itanium ia64
Opteron x86_64
Xeon with EM64T ia32e

Install the SFS client RPMs:

# rpm -ivh lustre-modules-*.architecture.rpm
# rpm -ivh lustre-lite-*.architecture.rpm
# rpm -ivh hpls-lustre-client-*.architecture.rpm
# rpm -ivh hpls-diags-client-*.architecture.rpm

Add the SFS system name to the /etc/hosts file. For example, if the SFS system is called south.my.domain.com, add an entry to the /etc/hosts file similar to the following:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
IP_address              south.my.domain.com south

#XC-CLUSTER Do Not Edit Below this Line
.
.
.

Create an /etc/sfstab.proto file on the head node to ensure the persistence of SFS mounts. The entries in this file use the following basic syntax:

ldap://sfs_server_name/filesystem mountpoint sfs optionlist 0 0

The /etc/sfstab.proto file enables you to customize the /etc/sfstab file on each node. The sfs service reads the /etc/sfstab.proto file as it starts and edits the /etc/sfstab file. In the /etc/sfstab.proto file you specify the content for the /etc/sfstab file for different node ranges (by inserting the #% directives).

Example 4-1 provides an example of an /etc/sfstab.proto file. See the HP StorageWorks Scalable File Share Client Installation and User Guide for details about the contents of this file; the mount options are very important.

In the example, n1044 is the head node, south is the name of the SFS system, and the #% directives specify that the content applies to the specified node names. The entries for the /hptc_cluster file system are required; all other lines in this example are optional and are provided for illustrative purposes only.

Do not mount the /hptc_cluster file system with the bg (background) mount option; the /hptc_cluster file system must be mounted in the foreground. As shown in the example, all other file systems are mounted in the foreground. The exception is the head node (n1044 ). Only the user file systems (/workspace and /apps ) are mounted on the head node using the bg mount option. Doing so means that the head node will boot even if there is a problem mounting these file systems

Example 4-1 Sample sfstab.proto File

#% n1044
ldap://south/hptc_cluster /hptc_cluster sfs nal=vib,max_cached_mb=128 0 0
ldap://south/workspace /workspace sfs nal=vib,max_cached_mb=1024,bg 0 0
ldap://south/apps /apps sfs nal=vib,max_cached_mb=256,bg 0 0

#% n[1039-1043]
ldap://south/hptc_cluster /hptc_cluster sfs nal=vib,max_cached_mb=128 0 0
ldap://south/workspace /workspace sfs nal=vib,max_cached_mb=1024,bg 0 0
ldap://south/apps /apps sfs nal=vib,max_cached_mb=256,bg 0 0

#% n[1-1038]
ldap://south/hptc_cluster /hptc_cluster sfs nal=vib,max_cached_mb=128,  \
    max_dirty_mb=2,max_rpcs_in_flight=2  0 0
ldap://south/workspace /workspace sfs nal=vib,max_cached_mb=1024 0 0
ldap://south/apps /apps sfs nal=vib,max_cached_mb=256,  \
    max_dirty_mb=2,max_rpcs_in_flight=2  0 0

Generate an /etc/sfstab file from the /etc/sfstab.proto file:
# service sfs gensfstab
Look at the contents of the /etc/sfstab file on the head node to ensure that it contains the content you expect.

Configure the interconnect and mount the SFS file system:

# /opt/hptc/etc/nconfig.d/C02network nconfigure `nodename`
# service sfs start

Make and Mount the /hptc_cluster File System Locally

If you made the decision to create a partition for the /hptc_cluster file system somewhere other than on the system disk on the head node, you must create and mount the /hptc_cluster file system now on a local disk and partition of your choice.

Remember to reformat the file system using the appropriate utility (such as mkfs) to clear out old data.

Create Local User Accounts (Optional)

If you intend to create local user accounts on your HP XC system rather than manage user accounts through another user authentication method (such as NIS or LDAP), use the Linux adduser command to create local user accounts on the system now, before the system is configured.

See the HP XC System Software Administration Guide if you need more information about creating local user accounts.

Special Considerations for Nagios and LSF

During the system configuration phase, the cluster_config command attempts to create a nagios and an lsfadmin account for use by Nagios and LSF, respectively.

To use existing nagios and lsfadmin user accounts from a site wide NIS system (or some other external user authentication system), you must manually create local XC accounts that mirror the site wide accounts (with matching user identification (UID) and group identification (GID) values). In that way, the cluster_config command uses these local accounts to properly configure Nagios and LSF with the right file permissions, access, and so on.

Override Default User and Group Account IDs (Optional)

Table 4-3 lists the user and group account IDs that are configured by default on an HP XC system. If any of these user and group identifiers conflict with other accounts or are not suitable for your environment, you can override them by creating the user accounts manually now (before running the cluster_config utility).

Table 4-3 HP XC User and Group Accounts

User Name	UID	Group Name	GID
`nagios`	`222`	`hpasm`	`222`
`qsnet`	`232`	`qsnet`	`232`
`slurm`	`500`	`slurm`	`500`
`lsfadmin`	`501`	`lsfadmin`	`501`

Customize Client Node Disk Partitioning (Optional)

The HP XC client node imaging process requires a single system disk on each client node for the operating system installation. Client node disk devices are partitioned automatically as part of the automated client imaging process.

Table 4-4 lists the default disk partition layout on client nodes. Partition sizes are created as a percentage of available disk space after subtracting the total swap size.

Table 4-4 Default Client Node Partition Layout

File System Name	Size
One swap partition	Swap space is calculated based on the amount of memory on the node, and it is governed by minimum and maximum values set by you By default, swap space is calculated as 100% x (total memory)
`/boot` or `/boot/efi`	1% x (total disk size minus swap space)
`/` (root)	60% x (total disk size minus swap space)
`/var`	39% x (total disk size minus swap space)

If the default partition layout is satisfactory for your needs, you do not have to do anything regarding client node disk partitioning.

However, if your system requires a different, customized partitioning layout on client disk devices, see Appendix E for instructions.