Serviceguard in VM Host Configuration

The following sections describe how to configure a VM Host as a Serviceguard node. In this configuration, if any of the resources used by a guest fail on the primary VM Host system, the guest fails over to an adoptive VM Host system, as illustrated in Figure 10-4.

Figure 10-4 Virtual Machine Failover to Another Cluster Member

To configure Serviceguard in Host:

Configure the Integrity VM multiserver environment, as described in Section .
Create the Serviceguard package, as described in Section .
Modify the Serviceguard package configuration files to match your guest environment, as described in Section .
Start the Serviceguard package, as described in Section .

Configuring the Integrity VM Multiserver Environment

The Integrity VM multiserver environment provides an integrated environment when guests are configured as Serviceguard packages. In a cluster that is configured as a multiserver environment, each VM Host is aware that the guests are Serviceguard packages and gives control of these distributed guests to ServiceGuard.

For example, two VM Hosts (host1 and host2) make up a Serviceguard cluster. A guest running on host1 is configured as a Serviceguard package that can fail over to host2. After you set up the multiserver environment, you can use the hpvmstatus command on each VM Host to display consistent guest package status information. When the guest package is running on host1, you can use the hpvmstatus command on host2 to display the guest's current status.

After you configure a guest as a Serviceguard package, you cannot use Integrity VM commands to start and stop the guest. Only Serviceguard commands control the packaged guest. This provides consistent control of the guests and also protects shared whole disk backing stores. Serviceguard ensures exclusive access for shared storage only when the disks are configured with a volume manager like LVM or VxVM. The Integrity VM multiserver environment extends this protection to whole disk backing stores and prevents accessing the same backing store from more than one guest at the same time.

The multiserver environment requires that Serviceguard be running on the VM Host, and allows you to configure guests as Serviceguard packages. The multiserver environment does not apply to configurations where Serviceguard is installed on the guest.

Set up the multiserver environment on each VM Host that is part of the multiserver environment, as follows:

Register each VM Host system that will be a member of the multiserver environment. Enter the following commands on each VM Host system:

# hpvmdevmgmt -a server:host-name
# hpvmdevmgmt -m server:host-name:attr:SERVERID=n
# hpvmdevmgmt -m server:host-name:attr:SERVERADDR=ip-address

Where:

host-name is the unqualified name of the VM Host system.
n is a unique number identifying this VM Host system in the multiserver environment. Enter a number from 1 through 255.
ip-addr is the IP address of the VM Host system. Use the IP address of the network connection that serves the multiserver environment.

Enter these commands on each VM Host system in the multiserver environment, specifying appropriate values. For example, to set up two nodes in the multiserver environment (host1 and host2), enter the following commands on host1:

# hpvmdevmgmt -a server:host1
# hpvmdevmgmt -m server:host1:attr:SERVERID=1 
# hpvmdevmgmt -m server:host1:attr:SERVERADDR=1.2.3.4
# hpvmdevmgmt -a server:host2
# hpvmdevmgmt -m server:host2:attr:SERVERID=2 
# hpvmdevmgmt -m server:host2:attr:SERVERADDR=1.2.3.5

Enter these same commands on host2.

Verify the registration by entering the following command on each member of the multiserver environment:

# hpvmdevmgmt -l server
host1:CONFIG=SERVER,SERVERADDR=16.116.9.0,SERVERID=1::WWID_NULL
host2:CONFIG=server,EXIST=NO,DEVTYPE=UNKNOWN,SHARE=NO,SERVERADDR=16.116.8.91,
SERVERID=2::WWID_NULL

The list of servers in the multiserver environment must match the list of nodes in the Serviceguard cluster configuration.

Creating Guest Packages

On the VM Host, create a package configuration file and control script for the guest using the following procedure:

Install Integrity VM and create the guest with all necessary virtual storage devices and vswitches. Repeat this procedure on each node in the multiserver environment.
Install, configure, and run HP Serviceguard on every node in the multiserver environment.
Configure the Integrity VM multiserver environment on the primary node and the alternate nodes as described in “Configuring the Integrity VM Multiserver Environment”.
Start the guest on the primary node using the hpvmstart command. Use the hpvmstatus command to verify the guest name and to make sure that it's running.

Create a Serviceguard package by running the hpvmsg_package script from the HP Serviceguard for Integrity VM Toolkit, which is installed in the /opt/cmcluster/toolkit/hpvm/ directory when you install Integrity VM. Specify the guest name as the argument to the command, as follows:

# /opt/cmcluster/toolkit/hpvm/hpvmsg_package.sh compass1

This is the HP Virtual Machine Serviceguard Toolkit Package Template Creation
script.

This script will assist the user develop and distribute a set of Serviceguard
package configuration template files and associated start, stop and monitor scripts.

The templates generated by these scripts will handle many guest configurations,
but it is only a template and may not be appropriate for your particular
configuration needs. You are encouraged to review and modify these template
files as needed for your particular environment.


Do you wish to continue? (y/n):y

[Virtual Machine Details]
Virtual Machine Name VM #  OS Type State     #VCPUs #Devs #Nets Memory  Runsysid
==================== ===== ======= ========= ====== ===== ===== ======= ========
compass1                 1 HPUX    Off            1     5     1  512 MB        0
[Storage Interface Details]
Guest                                 Physical
Device Adaptor    Bus Dev Ftn Tgt Lun Storage   Device
====== ========== === === === === === ========= =========================
disk   scsi         0   0   0   0   0 disk      /dev/rdsk/c12t0d0
disk   scsi         0   0   0   1   0 lv        /dev/vgsglvm/rlvol1
disk   scsi         0   0   0   2   0 file      /hpvm/g1lvm/hpvmnet2
disk   scsi         0   0   0   3   0 lv        /dev/vx/rdsk/sgvxvm/sgvxvms
disk   scsi         0   0   0   4   0 file      /hpvm/g1vxvm/hpvmnet2
disk   scsi         0   0   0   5   0 disk      /dev/rdsk/c12t0d5
[Network Interface Details]
Interface Adaptor    Name/Num   Bus Dev Ftn Mac Address
========= ========== ========== === === === =================
vswitch   lan        vswitch2     0   1   0 ea-5c-08-d3-70-f2
vswitch   lan        vswitch5     0   2   0 f2-c7-0d-09-ac-8f
vswitch   lan        vswitch6     0   4   0 92-35-ed-1f-6c-67

Would you like to create a failover package for this Virtual Machine summarized above? (y/n):y

Would you like to distribute the package to each cluster member? (y/n):y

The failover package template files for the Virtual Machine were successfully created.

The script asks you to confirm the following actions:

Creating a failover package
Distributing the package to all the cluster nodes

Respond to both prompts by entering y. The hpvm_package.sh script creates the virtual machine package template files in the etc/cmcluster/guest-name/ directory:

guest-name.config
guest-name.sh
hpvmsg_ctrl
hpvmsg_mon
hpvmsg_start
hpvmsg_stop

The hpvmsg_package is a utility that you can use to configure a guest as a Serviceguard package. The utility uses the guest name that you supply as an argument to create and populate the /etc/cmcluster/guest-name/ directory with a set of template files that contain basic Serviceguard parameter settings. HP recommends that you review and modify these template files as needed for your specific multiserver environment. For more information, see “Modifying the Package Configuration Files” and the Managing Serviceguard manual.

Stop the guest using the appropriate operating system command, or use the hpvmstop -F command on the VM Host system. (Because the guest has been configured as a Serviceguard package, the —F option is necessary.) For example:
# hpvmstop -P guest-name -F
Alternatively, enter the following command on the guest:
# /usr/sbin/shutdown -h now
Unmount all file backing stores and deactive any LVM logical volumes or deport VxVM volumes used as backing stores for the guests.
Verify that the package is set up correctly by entering the following command:
# cmcheckconf -v -C /etc/cmcluster/cluster-name.config -P /etc/cmcluster/guest-name/guest-name.config
Where:
- cluster-name is the name of the Serviceguard cluster.
- guest-name is the name of the guest.

Update and redistributed the binary configuration files to the /etc/cmcluster/guest-name/ directory on all cluster nodes:

# cmapplyconf -v -C /etc/cmcluster/cluster-name.config -P /etc/cmcluster/guest-name/guest-name.config

When prompted whether to modify the cluster configuration, enter y. For example:

# cmapplyconf -v -C /etc/cmcluster/cluster1.config  -P /etc/cmcluster/compass1/compass1.config

Checking cluster file: /etc/cmcluster/cluster.config
Checking nodes ... Done
Checking existing configuration ... Done
Gathering configuration information ... Done
Gathering configuration information ... Done
Gathering configuration information ..
Gathering storage information ..
Found 10 devices on node host1
Found 10 devices on node host2
Analysis of 20 devices should take approximately 3 seconds
0%----10%----20%----30%----40%----50%----60%----70%----80%----90%----100%
Found 7 volume groups on node charm
Found 7 volume groups on node clowder
Analysis of 14 volume groups should take approximately 1 seconds
0%----10%----20%----30%----40%----50%----60%----70%----80%----90%----100%
.....
Gathering Network Configuration ......... Done
Cluster cluster1 is an existing cluster
Parsing package file: /etc/cmcluster/compass1/compass1.config.
Package hpvmnet2 already exists. It will be modified.
Checking for inconsistencies .. Done
Cluster cluster1 is an existing cluster
Maximum configured packages parameter is 10.
Configuring 3 package(s).
7 package(s) can be added to this cluster.
200 access policies can be added to this cluster.
Modifying configuration on node host1
Modifying configuration on node host2

Modify the cluster configuration ([y]/n)? y
Marking/unmarking volume groups for use in the cluster
0%----10%----20%----30%----40%----50%----60%----70%----80%----90%----100%
Modifying the cluster configuration for cluster cluster1.
Modifying node host1 in cluster cluster1.
Modifying node host2 in cluster cluster1.
Modifying the package configuration for package compass1.
Completed the cluster creation.

If the package configuration file contains the appropriate settings, start the Serviceguard service as described in “Starting the Distributed Guest”.

Modifying the Package Configuration Files

The Serviceguard for Integrity VM toolkit creates templates that supply basic arguments to Serviceguard parameters. Review and modify the Serviceguard parameters based on the information for your Serviceguard cluster and the information supplied in the Managing Serviceguard manual. Make the appropriate changes to the guest-name.config and the guest-name.sh files.

Edit the package configuration file to add any LVM volume groups that are used by the distributed guest. Include a separate VOLUME_GROUP parameter for each cluster-aware volume group. These volume groups will be initialized with the cluster ID when the cmapplyconf command is used.

Starting the Distributed Guest

To start the distributed guest:, enter the following command:

# cmrunpkg -v guest-name

For example:

# cmrunpkg -v -p compass1
Running package compass1 on node host1.
cmrunpkg  : Successfully started package compass1.
cmrunpkg  : Completed successfully on all packages specified.

Verify that the guest is on and running. Use both the Integrity VM hpvmstatus command and the Serviceguard cmviewcl command to verify the status. For example:

# hpvmstatus -P compass1
[Virtual Machines]
Virtual Machine Name VM #  OS Type State     #VCPUs #Devs #Nets Memory  Runsysid
==================== ===== ======= ========= ====== ===== ===== ======= ========
compass1                 1 HPUX    On             1     5     1  512 MB        0

# cmviewcl -v compass1
CLUSTER      STATUS       
cluster1     up           

  NODE         STATUS       STATE        
  host1        up           running      

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME         
    PRIMARY      up           0/2/1/0/4/1         lan7         
    PRIMARY      up           0/2/1/0/6/1         lan9         
    PRIMARY      up           0/5/1/0/7/0         lan6         
    STANDBY      up           0/1/2/0             lan1         
    STANDBY      up           0/2/1/0/4/0         lan2         
    STANDBY      up           0/2/1/0/6/0         lan8         
    STANDBY      up           LinkAgg0            lan900       
    STANDBY      up           0/0/3/0             lan0         

    PACKAGE      STATUS       STATE        AUTO_RUN     NODE         
    compass1     up           running      disabled     host1     

      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual

      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS   NAME
      Service    up                  0         0   host1 

      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME                      
      Primary      up           enabled      host1     (current)       
      Alternate    up           enabled      host2    

  NODE         STATUS       STATE        
  host   2     up           running      

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME         
    PRIMARY      up           0/2/1/0/4/1         lan7         
    STANDBY      up           0/1/2/0             lan1         
    STANDBY      up           0/2/1/0/4/0         lan2         
    STANDBY      up           0/2/1/0/6/0         lan8         
    STANDBY      up           LinkAgg0            lan900       
    PRIMARY      up           0/5/1/0/7/0         lan6         
    PRIMARY      up           0/2/1/0/6/1         lan9         
    STANDBY      up           0/0/3/0             lan0

If desired, enter the cmmodpkg command to enable autorun and failover.

Starting the Vswitch Monitor

The vswitch monitor is responsible for monitoring the activities of the Serviceguard network monitor and for moving the vswitch configuration, when appropriate, between primary and standby network interfaces. The vswitch monitor requires no user configuration and is installed as part of the Integrity VM product. If Serviceguard is running and any distributed guests are configured, the vswitch monitor is automatically started on the VM Host system when the VM Host system boots. To start the vswitch monitor manually, use the following command:

# /sbin/init.d/vswitchmon start

To verify that the vswitch monitor is running, enter the following command:

# ps -ef | grep vswitchmon

Verifying That Distributed Guests Can Fail Over

To verify that the guests configured as Serviceguard packages and the multiserver environment are working properly, use the following commands to perform a manual failover:

On the original node (host1), verify that the package named compass1 is running:
host1# cmviewcl -v -p compass1

Halt the compass1 package on host1:

host1# cmhaltpkg compass1
Halting package compass1.

Start the package on the other VM Host system (host2):
host2# cmrunpkg -n host2 compass1
Enable the package:
host2# cmmodpkg -e compass1
On the adoptive node, verify that the compass1 package has started:
host2# cmviewcl -v -p compass1
On the adoptive node, verify that the guest named compass1 is on:
host2# hpvmstatus -P compass1

Managing Distributed Guests

To start, stop, and monitor distributed guests, use the Serviceguard commands described in this section. Do not use the Integrity VM commands (hpvmstart, hpvmstop, and hpvmmigrate) to manage distributed guests.

Starting Distributed Guests

To start a distributed guest, enter the following command:

# cmrunpkg guest-name

Stopping Distributed Guests

To stop a distributed guest, enter the following command:

# cmhaltpkg guest-name

Monitoring Distributed Guests

To monitor the distributed guest, enter the following command:

# cmviewcl -v -p guest-name

Modifying Distributed Guests

You can modify the resources for the distributed guest using the hpvmmodify command. However, if you modify the guest on one VM Host server, you must make the same changes on the other nodes in the multiserver environment.

After you modify vswitches, logical volumes, or file backing stores used by distributed guests, make sure that Serviceguard can continue to monitor the guests. Run the hpvmsg_package script and restart the guest packages to update the Serviceguard information.

Monitoring Network Connections

The vswitch monitor runs the vswitchmon.sh script on the VM Hosts in the multiserver environment and monitors the Serviceguard Network Manager by monitoring the syslog.log file. When it detects that Serviceguard is failing over a primary network to a standby network, the vswitch monitor halts, deletes, creates, and boots the vswitch associated with the primary network onto the the standby network. When the primary network is restored, Serviceguard and the vswitch monitor move the network and associated vswitch back to the primary network.