Managing HP Serviceguard for Linux,Fifth Edition

HP Part Number: B9903-90046

Edition:

Published: May 2005

Legal Notices

Table of Contents

Printing History

Preface

1 Serviceguard for Linux at a Glance

What is Serviceguard for Linux?

Failover

Viewing Serviceguard Clusters

Configuration Roadmap

2 Understanding Hardware Configurations for Serviceguard for Linux

Redundancy of Cluster Components

Redundant Network Components

Redundant Ethernet Configuration

Redundant Disk Storage

Supported Disk Interfaces
Disk Monitoring
Sample Disk Configurations

Redundant Power Supplies

3 Understanding Serviceguard Software Components

Serviceguard Architecture

Serviceguard Daemons

How the Cluster Manager Works

Configuration of the Cluster
Heartbeat Messages
Manual Startup of Entire Cluster
Automatic Cluster Startup
Dynamic Cluster Re-formation
Cluster Quorum to Prevent Split-Brain Syndrome
Cluster Lock
Use of a Lock LUN as the Cluster Lock
Use of the Quorum Server as a Cluster Lock
Types of Quorum Server Configuration

How the Package Manager Works

Configuring Packages
Deciding When and Where to Run and Halt Packages
Package Switching
Failover Policy
Failback Policy
Choosing Package Failover Behavior

How Package Control Scripts Work

What Makes a Package Run?
Before the Control Script Starts
During Run Script Execution
Normal and Abnormal Exits from the Run Script
Service Startup with cmrunserv
Disk Monitor Services
While Services are Running
When a Service or Subnet Fails
When a Package is Halted with a Command
During Halt Script Execution
Normal and Abnormal Exits from the Halt Script

How the Network Manager Works

Stationary and Relocatable IP Addresses
Adding and Deleting Relocatable IP Addresses
Bonding of LAN Interfaces
Bonding for Load Balancing
Remote Switching
ARP Messages after Switching

Volume Managers for Data Storage

Examples of Storage on Smart Arrays
Monitoring Disks
More Information on LVM

Responses to Failures

Transfer of Control (TOC) When a Node Fails
Responses to Hardware Failures
Responses to Package and Service Failures

4 Planning and Documenting an HA Cluster

General Planning

Serviceguard Memory Requirements
Planning for Expansion

Hardware Planning

SPU Information
LAN Information
Shared Storage
Disk I/O Information
Hardware Configuration Worksheet

Power Supply Planning

Power Supply Configuration Worksheet

Quorum Server Planning

Quorum Server Worksheet

Volume Manager Planning

Multipath Device Groups
Volume Groups and Physical Volume Worksheet

Cluster Configuration Planning

Quorum Server Information
Lock LUN Information
Cluster Configuration Parameters
Cluster Configuration Worksheet

Package Configuration Planning

Logical Volume and File System Planning
Planning for Expansion
Choosing Switching and Failover Behavior
Package Configuration File Parameters
Package Control Script Variables
Package Configuration Worksheet

5 Building an HA Cluster Configuration

Preparing Your Systems

Understanding the Location of Serviceguard Files
Enabling Serviceguard Command Access
Editing Security Files
Username Validation
Access Roles
Setting Access Controls for a Configured Clusters

Setting up the Quorum Server

Installing the Quorum Server
Running the Quorum Server
Creating a Package for the Quorum Server

Setting up the Lock LUN

Implementing Channel Bonding (Red Hat)

Sample Configuration
Restarting Networking
Viewing the Configuration

Implementing Channel Bonding (SLES9)

Creating the Logical Volume Infrastructure

Displaying Disk Information
Creating Partitions
Building Volume Groups for Use with Smart Array Cluster Storage (MSA 500 Series)
Building Volume Groups and Logical Volumes
Distributing the Shared Configuration to all Nodes
Testing the Shared Configuration
Storing Volume Group Configuration Data
Setting up Disk Monitoring

Configuring the Cluster

Using Serviceguard Manager to Configure the Cluster
Specifying a Quorum Server
Specifying a Lock LUN
Cluster Configuration Template File
Specifying Maximum Number of Configured Packages
Modifying Cluster Timing Parameters
Adding or Removing Nodes While the Cluster is Running
Verifying the Cluster Configuration
Cluster Lock Configuration Messages
Distributing the Binary Configuration File

Managing the Running Cluster

Checking Cluster Operation with Serviceguard Manager
Checking Cluster Operation with Serviceguard Commands
Setting up Autostart Features
Changing the System Message
Managing a Single-Node Cluster
Deleting the Cluster Configuration

6 Configuring Packages and Their Services

Using Serviceguard Manager to Configure a Package

Creating the Package Configuration

Creating the Package Configuration File

Writing the Package Control Script

Customizing the Package Control Script
Optimizing for Large Numbers of Storage Units
Configuring Disk Monitoring Services
Package Control Script Template File
Adding Customer Defined Functions to the Package Control Script
Adding or Removing Packages on a Running Cluster

Verifying the Package Configuration

Applying and Distributing the Configuration

Copying Package Control Scripts with Linux commands
Testing Cluster and Package Operation

Creating a Disk Monitor Configuration

Configuring All Disks for Monitoring
Configuring Disks on a Single Node for Monitoring

7 Cluster and Package Maintenance

Reviewing Cluster and Package States with the cmviewcl Command

Types of Cluster and Package States
Examples of Cluster and Package States

Using Serviceguard Manager

How Serviceguard Manager Works
Running Serviceguard Manager with a Command
Starting with a Specific Cluster
Starting Serviceguard Manager without a Specific Cluster
Connecting to an Object Manager
Opening a Saved File with Cluster Data
Viewing Cluster Data
Obtaining Help
Managing Cluster Objects

Viewing Status of Monitored Disks

Managing the Cluster and Nodes

Starting the Cluster When all Nodes are Down
Adding Previously Configured Nodes to a Running Cluster
Removing Nodes from Operation in a Running Cluster
Halting the Entire Cluster
Reconfiguring a Halted Cluster
Automatically Restarting the Cluster

Reconfiguring a Running Cluster

Adding Nodes to the Configuration While the Cluster is Running
Deleting Nodes from the Configuration While the Cluster is Running

Managing Packages and Services

Starting a Package
Halting a Package
Moving a Package
Reconfiguring a Package on a Halted Cluster

Reconfiguring a Package on a Running Cluster

Adding a Package to a Running Cluster
Deleting a Package from a Running Cluster
Changing Package Switching Behavior
Resetting the Service Restart Counter
Allowable Package States During Reconfiguration

Responding to Cluster Events

Single-Node Operation

Removing Serviceguard from a System

8 Troubleshooting Your Cluster

Testing Cluster Operation

Testing the Package Manager
Testing the Cluster Manager
Testing the Network Manager

Monitoring Hardware

Replacing Disks

Replacing a Faulty Mechanism in a Disk Array

Replacement of LAN Cards

Replacing a Failed Quorum Server System

Troubleshooting Approaches

Reviewing Package IP Addresses
Reviewing the System Log File
Reviewing Object Manager Log Files
Reviewing Configuration Files
Reviewing the Package Control Script
Using the cmquerycl and cmcheckconf Commands
Reviewing the LAN Configuration

Solving Problems

Serviceguard Command Hangs
Cluster Re-formations
System Administration Errors
Package Movement Errors
Node and Network Failures
Quorum Server Messages
Lock LUN Messages

A Serviceguard Commands

B Designing Highly Available Cluster Applications

Automating Application Operation

Insulate Users from Outages
Define Application Startup and Shutdown

Controlling the Speed of Application Failover

Replicate Non-Data File Systems
Evaluate the Use of a Journaled Filesystem (JFS)
Minimize Data Loss
Use Restartable Transactions
Use Checkpoints
Design for Multiple Servers
Design for Replicated Data Sites

Designing Applications to Run on Multiple Systems

Avoid Node Specific Information
Avoid Using SPU IDs or MAC Addresses
Assign Unique Names to Applications
Use uname(2) With Care
Bind to a Fixed Port
Bind to Relocatable IP Addresses
Give Each Application its Own Volume Group
Use Multiple Destinations for SNA Applications
Avoid File Locking

Restoring Client Connections

Handling Application Failures

Create Applications to be Failure Tolerant
Be Able to Monitor Applications

Minimizing Planned Downtime

Reducing Time Needed for Application Upgrades and Patches
Providing Online Application Reconfiguration
Documenting Maintenance Operations

C Integrating HA Applications with Serviceguard

Checklist for Integrating HA Applications

Defining Baseline Application Behavior on a Single System
Integrating HA Applications in Multiple Systems
Testing the Cluster

D Blank Planning Worksheets

Hardware Worksheet
Power Supply Worksheet
Quorum Server Worksheet
Mirror Group Worksheet
Volume Group and Physical Volume Worksheet
Cluster Configuration Worksheet
Package Configuration Worksheet
Package Control Script Worksheet

Index

List of Figures

1-1 Typical Cluster Configuration
1-2 Typical Cluster After Failover
1-3 Serviceguard Manager Cluster Status
1-4 Tasks in Configuring a Serviceguard Cluster
2-1 Redundant LANs
2-2 Mirrored Disks Connected for High Availability
3-1 Serviceguard Software Components on Linux
3-2 Lock LUN Operation
3-3 Quorum Server Operation
3-4 Quorum Server to Cluster Distribution
3-5 Package Moving During Failover
3-6 Before Package Switching
3-7 After Package Switching
3-8 Rotating Standby Configuration before Failover
3-9 Rotating Standby Configuration after Failover
3-10 CONFIGURED_NODE Policy Packages after Failover
3-11 Automatic Failback Configuration before Failover
3-12 Automatic Failback Configuration After Failover
3-13 Automatic Failback Configuration After Restart of Node 1
3-14 Package Time Line Showing Important Events
3-15 Package Time Line for Run Script Execution
3-16 Package Time Line for Halt Script Execution
3-17 Bonded Network Interfaces
3-18 Bonded NICs
3-19 Bonded NICs After Failure
3-20 Bonded NICs Configured for Load Balancing
3-21 Physical Disks Combined into LUNs
3-22 Smart Array Single Paths to LUNs
3-23 Smart Array LUNs Configured in Volume Groups
3-24 Physical Disks Combined into LUNs
3-25 Multiple Paths to LUNs Configured as Multipath Devices
3-26 Multiple Devices Configured in Volume Groups
4-1 Sample Cluster Configuration
4-2 Hardware Configuration Worksheet
4-3 Power Supply Configuration Worksheet
4-4 Quorum Server Worksheet
4-5 Volume Group and Physical Volume Worksheet
4-6 Cluster Configuration Worksheet
4-7 Package Configuration Worksheet
4-8 Package Control Script Worksheet
5-1 MSA500 Storage Volume Groups
7-1 Object Manager Login Screen
7-2 Serviceguard Manager Opening Screen
7-3 Object Manager Login Screen
7-4 Serviceguard Manager Open File Screen
7-5 Map and Cluster List Screen
7-6 Cluster Details Screen
7-7 Cluster Property Sheet Screen
7-8 Help Screen
7-9 Cluster Map Showing Action Menu
7-10 Cluster Startup Commands in Serviceguard Manager
7-11 Running Cluster
7-12 Display of Monitored Objects

List of Tables

1 Title not available (Printing History )
3-1 Package Configuration Data
3-2 Node Lists in Sample Cluster
3-3 Package Failover Behavior
3-4 Error Conditions and Package Movement
5-1 Changing Linux Partition Types
7-1 Serviceguard Manager Command Line Options
7-2 Types of Changes to Permanent Cluster Configuration
7-3 Types of Changes to Packages
A-1 Serviceguard Commands
D-1 Mirror Groups and Devices as Seen from Each Cluster Node

Managing HP Serviceguard for Linux,Fifth Edition

Technical documentation

» Table of Contents

» Index