United States-English

HP XC System Software : Administration Guide
Version 3.1

»

Technical documentation


	Complete book in PDF


»	Feedback

» Table of Contents

» Glossary

» Index

HP Part Number: 5991-7402

Published: November 2006

© Copyright 2003, 2004, 2005, 2006 Hewlett-Packard Development Company, L.P.

Abstract

This document describes the tools and procedures necessary to administer, monitor, and maintain an HP XC system running HP XC System Software Version 3.1.

Table of Contents

About This Document

Intended Audience
New and Changed Information in This Edition
Typographic Conventions
HP XC and Related HP Products Information
Related information
Manpages
HP Encourages Your Comments

1 HP XC Administration Environment

Understanding Nodes, Services, and Roles

Nodes
Services
Roles

File System

Key Operating System Directories
Log Files

HP XC Command Environment

HP XC Command Set
The nodelist Parameter
Executing a Command on Multiple Nodes

Configuration and Management Database

HP XC Configuration File Guidelines

Linux Configuration Files
Configuration Files
Configuration Files in Imaged Nodes

Installation and Software Distribution

Determining the Installation Type
Software Distribution

Improved Availability

Linux Virtual Server for HP XC Cluster Alias
Network Time Protocol
Network Address Translation
Network Information Service

Modulefiles

Administrator Passwords
Firewall
Secure Shell (ssh)

Recommended Administrative Tasks

2 Improved Availability

Purpose of the Availability Tool

Services Eligible for Improved Availability

Availability Sets

Determining Which Nodes Are in the Availability Set
Reconfiguring an Availability Set

HP Serviceguard Tasks

Viewing the Serviceguard Cluster Status
Moving Packages
Reimaging Nodes in the Availability Set

Transferring Control of Services

3 Starting Up and Shutting Down the HP XC System

Understanding the Node States

Starting Up the HP XC System

Starting All Nodes
Determining Which Nodes Require Imaging
Imaging and Starting Nodes
Restarting a Node for Imaging

Shutting Down the HP XC System

Shutting Down One or More Nodes

Determining a Node's Power Status

Locating a Given Node

Disabling and Enabling a Node

4 Managing and Customizing System Services

HP XC System Services

Displaying Services Information

Displaying All Services
Displaying the Nodes That Provide a Specified Service
Displaying the Services Provided by a Specified Node

Restarting a Service

Stopping a Service

Global System Services

Customizing Services and Roles

Overview of the HP XC Services Configuration
Service Configuration Sequence of Operation
Assigning Roles with the cluster_config Utility
The *config.d Directories
Configuration Scripts
Understanding Global Configuration Scripts
Advance Planning
Editing the roles_services.ini File
Creating a service.ini File
Adding a New Service
Verifying a New Service

5 Managing Licenses

License Manager and License File

Determining If the License Manager Is Running

Starting and Stopping the License Manager

Starting the License Manager
Stopping the License Manager
Restarting the License Manager

6 Managing the Configuration and Management Database

Accessing the Configuration and Management Database

Querying the Configuration and Management Database

Displaying Configuration Details
Displaying the Nodes That Provide a Specified Service
Displaying Blade Enclosure Information

Finding and Setting System Attribute Values

Backing Up the Configuration Database

Restoring the Configuration Database from a Backup File

Archiving Sensor Data from the Configuration Database

Restoring the Sensor Data from an Archive File

Purging Sensor Data from the Configuration and Management Database

Dumping the Configuration and Management Database

7 Monitoring the System

Monitoring Tools

Monitoring Strategy

Displaying System Environment Data

Monitoring Disks

Displaying System Statistics

Displaying System Sensors from the Command Line
Monitoring Processor Usage and Load from the Command Line
Monitoring Memory from the Command Line
Monitoring Paging and Swap Data from the Command Line

Logging Node Events

Understanding the Event Logging Structure
The syslog-ng.conf Rules File
Modifying the syslog-ng Rules Files

The collectl Utility

Running the collectl Utility from the Command Line
Running the collectl Utility as a Service
Running the collectl Utility in a Batch Job Submission

The netdump and crash Utilities

Installing Netdump and Crash
Configuring Netdump
Starting Netdump
Obtaining the Kernel Dump
Using the Crash Utility to Analyze a Kernel Dump File
Using the Crash Utility to Analyze a Live System

8 Monitoring the System with Nagios

Nagios Overview

Nagios Components
Nagios Hosts
Nagios Plug-Ins
Nagios Web Interface
Nagios Files

Using the Nagios Web Interface

Accessing the Nagios Web Interface
Using the Nagios Tactical Overview
Using the Nagios Service Detail View
Using the Nagios Service Problems View

Adjusting the Nagios Configuration

Stopping and Restarting Nagios
Updating the Nagios Configuration
Forwarding Nagios e-mail Alerts
Changing Sensor Thresholds
Adjusting the Time Allotted for Metrics Collection
Changing the Default Nagios User Name
Disabling Individual Nagios Plug-Ins

Configuring Nagios on HP XC Systems

Monitored Nagios Services
Nagios Default Settings
Understanding Nagios Alert Messages
System Event Log Monitoring

Nan Notification Aggregator and Delimiter

Nagios Report Generator Utility

9 Network Administration

Network Address Translation Administration
Network Time Protocol Service
Changing the External IP Address of a Head Node
Modifying Sendmail

10 Distributing Software Throughout the System

Overview of the Image Replication and Distribution Environment

Installing and Distributing Software Patches

Adding Software or Modifying Files on the Golden Client

Installing Additional RPMs from the HP XC System Software Installation DVD
Using File Overrides to the Golden Image
Using Per-Node Service Configuration

Determining Which Nodes Will Be Imaged

Golden Image Checksum

Updating the Golden Image

The cluster_config Utility
The updateimage Command
Exclusion Files
Ensuring That the Golden Image Is Current

Propagating the Golden Image to All Nodes

Using the Full Imaging Installation
Using the cexec Command

Maintaining a Global Service Configuration

11 Opening an IP Port in the Firewall

Opening Ports in the Firewall

Opening a Temporary Port in the Firewall
Opening an IP Port in the Firewall Persistently

12 Connecting to a Remote Console

Console Management Facility
Accessing a Remote Console

13 Managing Local User Accounts and Passwords

HP XC User and Group Accounts

General Procedures for Administering Local User Accounts

Adding a Local User Account

Modifying a Local User Account

Deleting a Local User Account

Configuring the ssh Keys for a User

Synchronizing the NIS Database

Changing Administrative Passwords

Changing the Superuser Password
Changing the CMDB Password
Changing the Interconnect Password
Changing the Console Port Password
Synchronizing the BMC/IPMI Password for CP6000 Systems
Changing the Nagios Administrator Password
Changing the LSF Administrator Password

14 Managing SLURM

Overview of SLURM

Configuring SLURM

Configuring SLURM System Interconnect Support
Configuring SLURM Servers
Configuring Nodes in SLURM
Configuring SLURM Partitions
Configuring SLURM Features
Propagating Resource Limits

Restricting User Access to Nodes

Job Accounting

Using the sacct Command
Disabling Job Accounting
Configuring Job Accounting

Monitoring SLURM

Draining Nodes

Configuring the SLURM Epilog Script

Maintaining the SLURM Daemon Log

Enabling SLURM to Recognize a New Node

Removing SLURM

15 Managing LSF

Standard LSF-HPC

LSF-HPC with SLURM

Integration of LSF-HPC with SLURM

Switching the Type of LSF Installed

LSF-HPC with SLURM Installation

LSF-HPC with SLURM Startup and Shutdown

Starting Up LSF-HPC with SLURM
Shutting Down LSF-HPC with SLURM

Controlling the LSF-HPC with SLURM Service

Launching Jobs with LSF-HPC with SLURM

Monitoring and Controlling LSF-HPC with SLURM Jobs

Job Accounting

LSF Daemon Log Maintenance

Load Indexes and Resource Information

LSF-HPC with SLURM Monitoring

LSF-HPC with SLURM Failover

Overview of LSF-HPC with SLURM Monitoring and Failover Support
Interplay of LSF-HPC with SLURM
Assigning the Resource Management Nodes
LSF-HPC with SLURM Failover and Running Jobs
Manual LSF-HPC with SLURM Failover

Enhancing LSF-HPC with SLURM

LSF-HPC with SLURM Enhancement Settings
Thresholds in LSF-HPC with SLURM and SLURM Interplay

Configuring an External Virtual Host Name for LSF-HPC with SLURM on HP XC Systems

16 Managing Modulefiles

17 Mounting File Systems

Overview of the Network File System on the HP XC System

Understanding the Global fstab File

Mounting Internal File Systems Throughout the HP XC System

Understanding the csys Utility in the Mounting Instructions
Mounting Internal File Systems

Mounting Remote File Systems

Understanding the Mounting Instructions
Mounting a Remote File System

18 Managing Software RAID Arrays

Overview of Software RAID

Software RAID-0
Software RAID-1

Installing Software RAID on the Head Node

Installing Software RAID on Client Nodes

Examining a Software RAID Array

Error Reporting

Removing Software RAID from Client Nodes

19 Using Diagnostic Tools

Using the sys_check Utility

Using the ovp Utility for System Verification

Using the dgemm Utility to Analyze Performance

Using the System Interconnect Diagnostic Tools

HP XC Diagnostic Tools for the Myrinet System Interconnect
Using Diagnostic Tools for the Quadrics System Interconnect
Using Diagnostic Tools for the InfiniBand Interconnect
Using Diagnostic Tools for the Gigabit Ethernet System Interconnect

20 Troubleshooting

General Troubleshooting

Mismatched Secure Shell Keys

Nagios Troubleshooting

Determining the Status of the Nagios Service
Nagios Fails to Start
Nagios Log Files
Running Nagios Plug-Ins Manually
Using the nrg Command's Analyze Mode

Messages Reported by Nagios

System Interconnect Troubleshooting

Myrinet System Interconnect Troubleshooting
Quadrics System Interconnect Troubleshooting
InfiniBand System Interconnect Troubleshooting

SLURM Troubleshooting

SLURM Configuration Issues
SLURM Run-Time Troubleshooting

LSF-HPC Troubleshooting

21 Servicing the HP XC System

Adding a Node

Replacing a Client Node

Replacing a System Interconnect Board in an CP6000 System

Software RAID Disk Replacement

Replacing a RAID Disk
Writing a Boot Block to the RAID Disk

A Installing LSF-HPC with SLURM into an Existing Standard LSF Cluster

Assumptions
Requirement
Sample Case
HP XC Preparation
Installing LSF-HPC with SLURM
Perform Post Installation Tasks
Configuring the LSF Alias
Starting LSF on the HP XC System
Sample Running Jobs
Troubleshooting

B Installing Standard LSF on a Subset of Nodes

Requirements
Assumptions
Sample Case
Instructions

C Setting Up MPICH

Downloading the MPICH Source Files
Building MPICH on the HP XC System
Running the MPICH Self-Tests
Installing MPICH

D HP MCS Monitoring

Customizing the Configuration for Your Installation
Regenerating the Nagios MCS Configuration
Useful Administrative Commands
MCS Log Files
Nagios Plug-Ins for MCS

List of Tables

1-1 Log Files
1-2 HP XC System Commands
1-3 HP XC Configuration Files
1-4 Recommended Administrative Tasks
3-1 Node States
4-1 Linux and third-party System Services
4-2 HP XC System Services
4-3 Components Used in Custom Services Configuration
4-4 Location of Configuration Script Directories
8-1 Supermon Metrics Collection Intervals
8-2 Monitored Nagios Services
8-3 Default Settings for Monitored Nagios Services
11-1 Default Open Internal and External Ports in the Firewall
11-2 Service Ports
12-1 Command-Line Mode Enter and Exit Key Sequences
13-1 HP XC User and Group Accounts
14-1 SLURM Configuration Settings
14-2 SLURM Partition Characteristics
14-3 Resource Limits
14-4 Output of the sinfo command for Various Transitions
15-1 LSF-HPC with SLURM Interpretation of SLURM Node States
15-2 Nagios Messages for LSF-HPC with SLURM Failover Monitor Status
15-3 Environment Variables for LSF-HPC with SLURM Enhancement (lsf.conf File)
15-4 Environment Variables for LSF-HPC with SLURM Enhancement (lsb.queues File)
17-1 NFS Daemon Ports
A-1 Host Names and IP Address Used in the Sample Case

List of Examples

4-1 Sample gconfig Script: Client Selection and Client-to-Server Assignment
4-2 Sample service.ini FIle
7-1 Using the collectl Utility from the Command Line
8-1 The nrg Utility System State Analysis
14-1 Using a SLURM Feature to Manage Multiple Node Types
15-1 Determining the LSF Execution Host
15-2 Comparison of Queues and the Configuration of the Job Starter Script
15-3 Basic Job Launch Without the JOB_STARTER Script Configured
15-4 Launching Another Job Without the JOB_STARTER Script Configured
15-5 Launching a Job Without the JOB_STARTER Script Using srun
15-6 Launching a Job Without the JOB_STARTER Script Using mpirun
15-7 Basic Job Launch with the JOB_STARTER Script Configured
17-1 Unedited fstab.proto File
17-2 The fstab.proto File Edited for Internal File System Mounting
17-3 The fstab.proto File Edited for Remote File System Mounting
20-1 Running a Nagios Plug-In from the Command Line
A-1 Running Jobs as a User on an External Node Launching to a Linux Itanium Resource
A-2 Running Jobs as a User on an External Node Launching to an HP XC Resource
A-3 Running Jobs as a User on an External Node Launching a Parallel Job
A-4 Running Jobs as a User on the HP XC Node Launching to a Linux Itanium Resource
A-5 Running Jobs as a User on the HP XC Node Launching to an HP XC Resource

Printable version


Privacy statement	Using this site means you accept its terms	Feedback to webmaster

© 2003 Hewlett-Packard Development Company, L.P.