Preparing the Cluster for Data Replication

This section assumes that you have already created one or more Serviceguard clusters for use in a disaster tolerant configuration. The following three sets of procedures will prepare Serviceguard clusters for use with Continuous Access XP data replication in a metropolitan or continental cluster.

Creating the RAID Manager Configuration
Defining Storage Units
Configuring Packages

Creating the RAID Manager Configuration

Use these steps to create the configuration:

Ensure that the XP Series disk arrays are correctly cabled to each host system that will run packages whose data reside on the arrays.
Each XP Series disk array must be configured with redundant Continuous Access links, each of which is connected to a different LCP or RCP card. To prevent a single point of failure (SPOF), there must be at least two physical boards in each XP for the Continuous Access links. Each board usually has multiple ports. However, a redundant Continuous Access link must be connected to a port on a different physical board from the board that has the primary Continuous Access link. When using bi-directional configurations, where data center A backs up data center B and data center B backs up data center A, you must have at least four Continuous Access links, two in each direction. Four Continuous Access links are also required in uni-directional configurations in which you want to allow failback.
Install the Raid Manager XP software on each host system that has data residing on the XP disk arrays.
Edit the /etc/services file, adding an entry for the Raid Manager instance to be used with the cluster.
The format of the entry is:
horcm<instance-number> <port-number>/udp
For example:
horcm0 11000/udp #Raid Manager instance 0
For more detail, see the /opt/cmcluster/toolkit/SGCA/Samples/services.example file.
Use the ioscan command to determine what devices on the XP disk array have been configured as command devices. The device-specific information in the right most column of the ioscan output will have the suffix -CM for these devices; for example, OPEN-3-CM.
If there are no configured command devices on the disk array, you must create two before proceeding. Each command device must have alternate links (PVLinks). The first command device is the primary command device. The second command device is a redundant command device and is used only upon failure of the primary command device. The command devices must be mapped to the various host interfaces by using the SVP (disk array console) or a remote console.
Copy the default Raid Manager configuration file to an instance-specific name.
# cp /etc/horcm.conf /etc/horcm0.conf
Create a minimum Raid Manager configuration file by editing the following fields in the file created in the previous step:
- HORCM_MON—enter the host-name of the system on which you are editing and the TCP/IP port number specified for this Raid Manager instance in the /etc/services file.
- HORCM_CMD—enter the primary and alternate link device file names for both the primary and redundant command devices (for a total of four raw device file names).
  CAUTION: Make sure that the redundant command device is NOT on the same physical device as the primary command device. Also, make sure that it is on a different bus inside the XP series disk array.
If the Raid Manager protection facility is enabled, set the HORCPERM environment variable to the pathname of the HORCM permission file, then export the variable.
# export HORCMPERM=/etc/horcmperm0.conf
If the Raid Manager protection facility is not used or disabled, export the HORCPERM environment variable.
# export HORCMPERM=MGRNOINST
Start the Raid Manager instance by using horcmstart.sh <instance-#>.
# horcmstart.sh 0
Export the environment variable that specifies the Raid Manager instance to be used by the Raid Manager commands. For example, with the POSIX shell type.
# export HORCMINST=0
Now, use Raid Manager commands to get further information from the disk arrays.
To verify the software revision of the Raid Manager and the firmware revision of the XP disk array.
# raidqry -l
NOTE: Check for the minimum requirement level for XP, Raid Manager software, and firmware for your version listed in the Metrocluster Continuous Access XP Release Notes.
To view a list of the available devices on the disk arrays.
# raidscan
The above command must be invoked separately for each host interface connection to the disk array. For example, if there are two Fibre Channel host adapters.
# raidscan -p CL1-A
# raidscan -p CL1-B
NOTE: There must also be alternate links for each device, and these alternate links must be on different busses inside the XP disk array. These alternate links, for example, may be CL2-E and CL2-F.
Unless the devices have been previously paired either on this or another host, the devices will show up as SMPL (simplex). Paired devices will show up as PVOL (primary volume) or SVOL (secondary volume).
XP arrays (XP 10000/XP 12000 and beyond) support external attached storage devices to be configured as either P-VOL or S-VOL or both of a Continuous Access pair. From a Continuous Access perspective, there is no difference between a pair created from internal devices and a pair created on external devices.
Refer to the HP StorageWorks XP documentation for information on the configuration requirements of external storage devices attached to XP arrays and supported external storage devices.
Determine which devices will be used by the application package. Define a device group that contains all of these devices. It is recommended that you use a name that is easily associated with the package. For example, a device group name of “db-payroll” is easily associated with the database for the payroll application. A device group name of “group1” would be more difficult to relate to an application.
Edit the Raid Manager configuration file (horcm0.conf) in the above example to include the devices and device group used by the application package. Only one device group may be specified for all of the devices that belong to a single application package. These devices are specified in the field HORCM_DEV.
Also complete the HORCM_INST field, supplying the names of only those hosts that are attached to the XP disk array that is remote from the disk array directly attached to this host. For example, with the continental cluster shown in Figure 3-4 “Disaster Tolerant Cluster” (node 1 and node 2 in the primary cluster and nodes 3 and node 4 in the recovery cluster), you would specify only nodes 3 and node 4 in the HORCM_INST field in a file you are creating on node 1 on the primary cluster. Node 1 would have previously been specified in the HORCM_MON field.
Figure 3-4 Disaster Tolerant Cluster
As an example, see file /opt/cmcluster/toolkit/SGCA/Samples/horcm0.conf.<sys-name>
Restart the Raid Manager instance so that the new information in the configuration file is read.
# horcmshutdown.sh <instance-#>
# horcmstart.sh <instance-#>
Repeat these steps on each host that will run this particular application package. If a host may run more than one application package, you must incorporate device group and host information for each of these packages.
Note that the Raid Manager configuration file must be different for each host, especially for the HORC_MON and HORC_INST fields.
If not previously done, create the paired volumes.
# paircreate -g <devgroup> -f <fencelevel> -vl -c15
This command must be issued before creating volume groups. For creating a pair of journal groups, refer to the next section, “Pair Creation of Journal Groups”.
CAUTION: Paired devices must be of compatible sizes and types.
When using the paircreate command to create PVOL/SVOL Continuous Access pairs, specify the -c 15 switch to ensure the fastest data copy from PVOL to SVOL.

Pair Creation of Journal Groups

The Continuous Access Journal has the same characteristic as Continuous Access Asynchronous such that Raid Manager controls Continuous Access Journal similar to Continuous Access Asynchronous.

The Raid Manager configuration of the device group pair for Continuous Access Journal is exactly the same as the configuration of the Continuous Access Asynchronous device group pair. In the /etc/horcm0.conf file, do not specify any journal volumes or journal group number. Only data volumes (device group and it’s devices) need to be in the configuration file.

Creating Continuous Access Journal Pair

To create a journal group pair, use the paircreate command.

# paircreate -g <device_group> -f async -vl -c 15 -jp <id> \ -js <id>

Similar to Continuous Access Asynchronous, the fence “async” must be assigned to the command with two additional options -jp and -js.

-jp <id> : This option is to specify a journal group ID for PVOL

-js <id> : This option is to specify a journal group ID for SVOL

The -jp and -js options are required if the device group is configured to use Continuous Access Journal. The <id> used with -jp and -js option do not need to be the same.

Sample Raid Manager Configuration File

The following is an example of a Raid Manager configuration file for one node (ftsys1).

#
# horcm0.conf.ftsys1
#    - This is an example Raid Manager configuration file for node ftsys1.
#      Note that this configuration file is for Raid Manager instance 0,
#      which can be determined by the "0" in the filename "horcm0.conf".
#
 
# Whenever this configuration file is changed, you must stop and restart the
# instance of Raid Manager before the changes will be recognized.  This can be done using the following commands:
#
# horcmshutdown.sh <instance>
# horcmstart.sh <instance>
#
# After restarting the Raid Manager instance, you should confirm that there
# are no configuration errors reported by running the pairdisplay command
# with the "-c" option.
#
# NOTE: The Raid Manager command device (RORCM_CMD) cannot be used for
#       data storage (it is reserved for private Raid Manager usage).
 
#/************************ HORCM_MON *************************************/
#
# The HORCM_MON parameter is used for monitoring and control of device groups
# by the Raid Manager.
# It is used to define the IP address, port number, and paired volume error
# monitoring interval for the local host.
# <ip_address>
#  Defines a network address used by the local host. This can be a host name
#  or an IP address.
# <service>
#  Specifies the port name assigned to the Raid Manager communication path,
#  which is must also be defined in /etc/services.  If a port number, rather
#  than a port name is specified, the port number will be used.
# <poll_interval>
#  Specifies the interval used for monitoring the paired volumes.  By
#  increasing this interval, the Raid Manager daemon load is reduced.
#  If this interval is set to -1, the paired volumes are not monitored.
# <timeout>
#  Specifies the time-out period for communication with the Raid Manager
#  server.
 
HORCM_MON
#ip_address     service         poll_interval(10ms)     timeout(10ms)
ftsys1          horcm0          1000                    3000
 
#/************************* HORCM_CMD *************************************/
#
# The HORCM_CMD parameter is used to define the special files (raw device
# file names) of the Raid Manager command devices used for the monitoring
# and control of Raid Manager device groups.
# Define the special device files corresponding to two or more command devices
# in order to use the Raid Manager alternate command device feature.  An
# alternate command device must be configured, otherwise a failure of a
# single command device could prevent access to the device group.
# Each command device must have alternate links (PVLinks).  The first command
# device is the primary command device.  The second command device is a
# redundant command device and is used only upon failure of the primary
# command device.  The command devices must be mapped to the various host
# interfaces by using the SVP (disk array console) or a remote console.
 
HORCM_CMD
#Primary            Primary Alt-Link    Secondary           Secondary Alt-link
#dev_name           dev_name            dev_name            dev_name
/dev/rdsk/c4t1d0    /dev/rdsk/c5t1d0    /dev/rdsk/c4t0d1    /dev/rdsk/c5t0d1
 
#/************************* HORCM_DEV *************************************/
#
# The HORCM_DEV parameter is used to define the addresses of the physical
# volumes corresponding to the paired logical volume names.  Each group
# name is a unique name used by the hosts which will access the volumes.
#
# The group and paired logical volume names defined here must be the same for
# all other (remote) hosts that will access this device group.
# The hardware SCSI bus, SCSI-ID, and LUNs for the device groups do not need
# to be the same on remote hosts.
#
# <dev_group>
#  This parameter is used to define the device group name for paired logical
#  volumes.  The device group name is used by all Raid Manager commands for
#  accessing these paired logical volumes.
# <dev_name>
#  This parameter is used to define the names of the paired logical volumes
#  in the device group.
# <port#>
#  This parameter is used to define the XP256 port number used to access the
#  physical volumes in the XP256 connected to the "dev_name".  Consult your
#  XP256 for valid Port numbers to specify here.
# <TargetID>
#  This parameter is used to define the SCSI target ID of the physical
#  volume on the port specified in "port#".
# <LUN#>
#  This parameter is used to define the SCSI logical unit number (LUN) of
#  the physical volume specified in "targetID".
 
HORCM_DEV
#dev_group      dev_name        port#           TargetID        LUN#
pkgA            pkgA_index      CL1-E           0               1
pkgA            pkgA_tables     CL1-E           0               2
pkgA            pkgA_logs       CL1-E           0               3
pkgB            pkgB_d1         CL1-E           0               4
pkgC            pkgC_d1         CL1-E           0               5
pkgD            pkgD_d1         CL1-E           0               2
 
#/************************* HORCM_INST ************************************/
#
# This parameter is used to define the network address (IP address or host
# name) of the remote hosts which can provide the remote Raid Manager access
# for each of the device group secondary volumes.
# The remote Raid Manager instances are required to get status or provide
# control of the remote devices in the device group.  All remote hosts
# must be defined here, so that the failure of one remote host will prevent
# obtaining status.
#
# <dev_group>
#  This is the same device group names as defined in dev_group of HORC_DEV.
# <ip_address>
#  This parameter is used to define the network address of the remote hosts
#  with Raid Manager access to the device group.   This can be either an
#  IP address or a host name.
# <service>
#  This parameter is used to specify the port name assigned to the Raid
#  Manager instance, which must be registered in /etc/services.  If this is
#  a port number rather than a port name, then the port number will be used.
 
HORCM_INST
#dev_group      ip_address      service
pkgA            ftsys1a         horcm0
pkgA            ftsys2a         horcm0
pkgB            ftsys1a         horcm0
pkgB            ftsys2a         horcm0
pkgC            ftsys1a         horcm0
pkgC            ftsys2a         horcm0
pkgD            ftsys1a         horcm0
pkgD            ftsys2a         horcm0

Notes on the Raid Manager Configuration

A single XP device group must be defined for each package on each host that is connected to the XP series disk array. Device groups are defined in the Raid Manager configuration file under the heading HORCM_DEV. The disk target IDs and LUNs for all Physical Volumes (PVs) defined in Volume Groups (VGs) that belong to the package must be defined in one XP device group on each host system that may ever run one or more Continentalclusters packages. The device group name (dev_group) is user-defined and must be the same on each host in the continental cluster that accesses the XP disk array. The device group name (dev_group) must be unique within the cluster; it should be a name that is easily associated with the application name or Serviceguard package name.

The TargetID and LU# fields for each device name may be different on different hosts in the clusters, to allow for different hardware I/O paths on different hosts. See the sample convenience scripts in the Samples directory included with this toolkit for examples.

Configuring Automatic Raid Manager Startup

After editing the Raid Manager configuration files and installing them on the nodes that are attached to the XP Series disk arrays, you should configure automatic Raid Manager startup on the same nodes. This is done by editing the rc script /etc/rc.config.d/raidmgr. Set the START_RAIDMGR parameter to 1, and define RAIDMGR_INSTANCE as the number of the Raid Manager instances being used. By default, this is zero (0).

An example of the edited startup file is shown below:

#*************************** RAIDMANAGER *************************
 
# Metrocluster with Continuous Access Toolkit script for configuring the
# startup parameters for a HP StorageWorks Disk Array XP Raid Manager
# instance.  The Raid Manager instance must be running before any
# Metrocluster package can start up successfully.
#
# @(#) $Revision: 1.8 $
#
# START_RAIDMGR:        If set to 1, this host will attempt to start up
#                       an instance of the Disk Array XP Raid Manager,
#                       which must be running before a Metrocluster package
#                       can be successfully started.  If set to 0, this host
#                       will not attempt to start the Raid Manager.
#
# RAIDMGR_INSTANCE      This is the instance number of the Raid Manager
#                       instance to be started by this script.  The instance
#                       number specified here must be the same as the
#                       instance number specified in the Metrocluster
#                       package control script.
#                       Consult your Raid Manager documentation for more
#                       information on Raid Manager instances.
#
# See the Metrocluster and Raid Manager documentation for more information
# on configuring this script.
#
START_RAIDMGR=0
RAIDMGR_INSTANCE=0

Defining Storage Units

Both LVM and VERITAS VxVM storage can be used in disaster tolerant clusters. The following sections show how to set up each type:

Creating and Exporting LVM Volume Groups using Continuous Access XP

Use the following procedure to create and export volume groups:

Define the appropriate Volume Groups on each host system that might run the application package.
# mkdir /dev/vgxx
# mknod /dev/vgxx/group c 64 0xnn0000
where the name /dev/vgxx and the number nn are unique within the entire cluster.
Create the Volume Group only on the primary system on the primary cluster. Use the vgcreate and perhaps the vgextend command, specifying the appropriate special device file names. See the sample script /opt/cmcluster/toolkit/SGCA/Samples/mk1VGs.
Create the logical volume(s) for the volume group.
Export the Volume Groups on the primary system without removing the special device files.
# vgchange -a n <vgname>
Make sure that you copy the mapfiles to all of the host systems.
# vgexport -s -p -m <mapfilename> <vgname>
On the primary cluster import the VGs on all of the other systems that might run the Serviceguard package and backup the LVM configuration.
# vgimport -s -m <mapfilename> <vgname>
# vgchange -a y <vgname>
# vgcfgbackup <vgname>
# vgchange -a n <vgname>
See the sample script /opt/cmcluster/toolkit/SGCA/Samples/mk2imports.
On the recovery cluster import the VGs on all of the systems that might run the Serviceguard recovery package and backup the LVM configuration.
# pairsplit -g <dev_name> -rw
# vgimport -s -m <mapfilename> <vgname>
# vgchange -a y <vgname>
# vgcfgbackup <vgname>
# vgchange -a n <vgname>
# pairresync -g <dev_name> -c 15
See the sample script /opt/cmcluster/toolkit/SGCA/Samples/mk2imports.Skip the pairsplit/pairresync, however this will not activate the volume group to perform the vgcfgbackup. Perform the vgcfgbackup when the volume group is activated during the first recovery package activation.
When using the pairresync command to resynchronize PVOL/SVOL Continuous Access pairs, specify the -c 15 switch to ensure the fastest resynchronization which reduces the vulnerability of a rolling disaster.

Creating VxVM Disk Groups using Continuous Access XP

If using VERITAS storage, use the following procedure to create disk groups. It is assumed a VERITAS root disk (rootdg) has already been created on the system where configuring the storage. The following section shows how to set up VERITAS disk groups. On one node do the following:

Create the device pair to be used by the package.
# paircreate -g devgrpA -f never -vl -c 15
Check to make sure the devices are in the PAIR state.
# pairdisplay -g devgrpA
Initialize disks to be used with VxVM by running the vxdisksetup command only on the primary system.
# vxdisksetup -i c5t0d0
Create the disk group to be used with the vxdg command only on the primary system.
# vxdg init logdata /dev/dsk/c5t0d0
Verify the configuration.
# vxdg list
Use the vxassist command to create the logical volume.
# vxassist -g logdata make logfile 2048m
Verify the configuration.
# vxprint -g logdata
Make the filesystem.
# newfs -F vxfs /dev/vx/rdsk/logdata/logfile
Create a directory to mount the volume group.
# mkdir /logs
Mount the volume group.
# mount /dev/vx/dsk/logdata/logfile /logs
Check if file system exits, then unmount the file system.
# umount /logs

Validating VxVM Disk Groups using Metrocluster/Continuous Access Data Replication

The following section describes how to validate the VERITAS disk groups on one node:

Deport the disk group.
# vxdg deport logdata
Enable other cluster nodes to have access to the disk group.
# vxdctl enable
Suspend the Continuous Access link and have SVOL Read/Write permission.
# pairsplit -g devgrpA -rw
Import the disk group.
# vxdg -tfC import logdata
Start the logical volume in the disk group.
# vxvol -g logdata startall
Create a directory to mount the volume.
# mkdir /logs
Mount the volume.
# mount /dev/vx/dsk/logdata/logfile /logs
Check to make sure the file system is present, then unmount the file system.
# umount /logs
Resynchronize the Continuous Access pair device.
# pairresync -g devicegroupname -c 15