Jump to content United States-English
HP.com Home Products and Services Support and Drivers Solutions How to Buy
» Contact HP
More options
HP.com home
 Designing Disaster Tolerant High Availability Clusters: > Chapter 6 Building Disaster Tolerant Serviceguard Solutions Using Metrocluster with EMC SRDF

Building a Metrocluster Solution with EMC SRDF
» 

Technical documentation

Complete book in PDF
» Feedback
Content starts here

 » Table of Contents

 » Glossary

 » Index

spacerspacerspacer
spacer
spacer
  		 
 

Setting up 1 by 1 Configurations

The most common Symmetrix configuration used with Metrocluster/SRDF is a 1 by 1 configuration in which there is a single Symmetrix frame at each Data Center. This section describes how to set up this configuration using SymCLI and HP-UX commands. It is assumed that you have already set up the Symmetrix CLI database on each node as described in the previous section, ““Preparing the Cluster for Data Replication”.”

A basic 1 by 1 configuration is shown in Figure 6-6 “Mapping HP-UX Device File Names to Symmetrix Units”, which is a graphical view of the data in Table 6-2 “Partial Mapping for a 4 Node Cluster”.

Figure 6-6 Mapping HP-UX Device File Names to Symmetrix Units

Mapping HP-UX Device File Names to Symmetrix Units

Creating Symmetrix Device Groups

A single Symmetrix device group must be defined for each package on each node that is connected to the Symmetrix. The following procedure must be done on each node that may potentially run the package:

NOTE: The sample scripts mk3symgrps.nodename can be modified to automate these steps.

  1. Use the symdg command, or modify the mk3symgrps.nodename script to define an R1 and an R2 device group for each package:

    # symdg create -type RDF1 devgroupname

    Issue this command on nodes attached to the R1 side.

    # symdg create -type RDF2 devgroupname

    Issue this command on nodes attached to the R2 side.

    The group name must be the same on each node on the R1 and R2 side. The devgroup name used will be later placed in variable DEVICE_GROUP defined in pkg.env file

  2. Use the symld command to add all LUNs that comprise the Volume Group for that package on that host. The HP-UX device file names for all Volume Groups that belong to the package must be defined in one Symmetrix device group. All devices belonging to Volume Groups that are owned by an application package must be added to a single Symmetrix device group

    # symld -g devgroupname add dev devnumber1

    # symld -g devgroupname add dev devnumber2

    This is where Table 6-2 “Partial Mapping for a 4 Node Cluster” is helpful. Although the name of the HP-UX device file names on each node specified may be different, the device group must be the same on each node.

    When creating the Symmetrix device groups, you may specify only one HP-UX path to a particular Symmetrix device. Do not specify alternate paths (PVLinks). The SymCLI uses the HP-UX path only to determine to which Symmetrix device you are referring. The Symmetrix device may be added to the device group only once.

    NOTE: Symmetrix Logical Device names must be the default names of the form DEVnnn (e.g., DEV001). Do not use the option for creating your own device names.

The script must be customized for each system including:

  • particular HP-UX device file names

  • Symmetrix device group name (arbitrary, but unique name may be chosen for each group that defines all of the volume groups (VGs) belonging to a particular Serviceguard package).

  • keyword RDF1 or RDF2

Configuring Gatekeeper Devices

NOTE: The sample scripts mk4gatekpr.nodename can be modified to automate these steps.

Gatekeeper devices must be unique per Serviceguard package to prevent contention in the Symmetrix when commands are issued, such as two or more packages starting up at the same time. Gatekeeper devices are unique to a Symmetrix unit. They are not replicated across the SRDF link. Gatekeeper devices are marked GK in the syminq output, and are usually 2880 KB in size.

  1. Define at least two gatekeepers per package per node (assuming PV links are used). They will only be available for use by that node. Each gatekeeper device is configured on different physical links:

    # symgate -sid sidnumber1 define dev devnumber1

    # symgate -sid sidnumber2 define dev devnumber2

  2. Associate the gatekeeper devices with the Symmetrix device group for that package:

    # symgate -sid sidnumber1 -g devgroupname \
    associate dev devnumber1

    # symgate -sid sidnumber2 -g devgroupname \
    associate dev devnumber2

  3. Define a pool of four or more additional gatekeeper devices that are not associated with any particular node. The SymCLI will switch to an alternate gatekeeper device if the path to the primary gatekeeper device fails. Certain software requires the existence of gatekeepers: OSM graphical interface and SymCLI. These gatekeeper devices are seen from all of the nodes.

Verifying the EMC Symmetrix Configuration

When you are finished with all these steps, use the symrdf list command to get a listing of all devices and their states.

You may want to back up the SymCLI database on each node so that you do not have to do these configuration steps again if a failure corrupts the database. The SymCLI database is a binary file in the directory /var/symapi/db.

Creating and Exporting Volume Groups

Use the following procedure to create volume groups and export them for access by other nodes. The sample script mk1VGs in the /opt/cmcluster/toolkit/SGSRDF/Samples directory can be modified to automate these steps.

  1. Define the appropriate Volume Groups on each node that might run the application package. Use the commands:

    # mkdir /dev/vgxx

    # mknod /dev/vgxx/group c 64 0xnn0000

    where the name /dev/vgxx and the number nn are unique within the cluster.

  2. Create volume groups only on the primary system. Use the vgcreate and the vgextend command, specifying the appropriate HP-UX device file names.

    # vgcreate vgname /dev/dsk/cxtydz

  3. Use the vgchange command to de-activate the volume group and use the vgexport command with the -p option to export the VGs on the primary system without removing the HP-UX device files:

    # vgchange -a n vgname

    # vgexport -v -s -p -m mapfilename vgname

    Make sure that you copy the map files to all of the nodes. The sample script Samples/ftpit shows a semi-automated way (using ftp) to copy the files. You need only enter the password interactively.

Importing Volume Groups on Other Nodes

Use the following procedure to import volume groups. The sample script mk2imports can be modified to automate these steps.

  1. Import the VGs on all of the other systems that might run the Serviceguard package and backup the LVM configuration. Make sure that you split the logical SRDF links before importing the VGs, especially if you are importing the VGs on the R2 side. Use the commands:

    # symrdf -g devgrpname split -v

    # vgimport -v -s -m mapfilename vgname

  2. Back up the configuration. Use the following commands:

    # vgchange -a y vgname

    # vgcfgbackup vgname

    # vgchange -a n vgname

    # symrdf -g devgrpname establish -v

    See the sample script Samples/mk2imports.

NOTE: Exclusive activation must be used for all volume groups associated with packages that use the EMC. The design of Metrocluster/SRDF assumes that only one system in the cluster will have a VG activated at a time.

Configuring PV Links

The examples in the previous sections show the use of the vgimport and vgexport commands with the -s option. Also, the mk1VGs script uses a -s in the vgexport command, and the mk2imports script uses a -s in the vgimport command. You may wish to remove this option from both commands if you are using PV links. The -s option to the vgexport command saves the volume group id (VGID) in the map file, but it does not preserve the order of PV links. To specify the exact order of PV links, do not use the -s option with vgexport, and in the vgimport command, enter the individual links in the desired order, as in the following example:

# vgimport -v -m mapfilename vgname linkname1 linkname2

Grouping the Symmetrix Devices at Each Data Center

The use of R1/R2 devices in M by N configurations of multiple Symmetrix frames is enabled by means of consistency groups. A consistency group is a set of Symmetrix RDF devices that are configured to act in unison to maintain the integrity of a database. Because Metrocluster/SRDF works at the device group level, the consistency group is implemented and managed as a single device group even though it spans multiple Symmetrix frames. Consistency groups are managed by the EMC PowerPath product. When PowerPath is installed, the Symmetrix tracks the I/Os that are written to the devices in the consistency group. If an I/O cannot be written to a remote Symmetrix due to a failure of a remote device or an RDF link, the data flow to the other Symmetrix will be halted in less than one second. Once mirroring is resumed, any updates will propagate with normal SRDF operation.

Figure 6-7 “2 X 2 Node and Data Center Configuration with Consistency Groups” shows that when there is a break in the links between two of the Symmetrix frames, the use of consistency groups (dashed oval lines) ensures that the other two links are also suspended.

Figure 6-7 2 X 2 Node and Data Center Configuration with Consistency Groups

2 X 2 Node and Data Center Configuration with Consistency Groups

Setting up M by N Configurations

Metropolitan clusters using EMC/SRDF can be built in configurations that use more than two EMC Symmetrix disk arrays. In such configurations, M arrays located in Data Center A may be connected to N arrays located in Data Center B. This section describes how to set up this configuration using SymCLI and HP-UX commands. It is assumed that you have already set up the Symmetrix CLI database on each node as described in the previous section, “Preparing a Cluster for Metrocluster/SRDF.” It is also assumed that Symmetrix PowerPath software is installed on all nodes.

WARNING! M by N configurations cannot be used with R1/R2 swapping.

Figure 6-8 “Devices and Symmetrix Units in M by N Configurations” shows a 2 by 2 configuration. Data in this figure are used in the example commands given in the following sections. The example shows R1 devices at one data center and R2 devices with Business Continuity Volumes (BCVs) at the other, but a bidirectional configuration is also possible, with R1 devices on both sites.

Figure 6-8 Devices and Symmetrix Units in M by N Configurations

Devices and Symmetrix Units in M by N Configurations

Creating Symmetrix Device Groups

For each node on the R1 side (node1 and node2), create the device groups as follows. You have to create two device groups because device groups do not span frames.

NOTE: In a 1 by 2 or 2 by 1 configuration, on either the R1 or R2 side, whichever side has the one Symmetrix frame, a minimum of two device groups is required. The rule of thumb is that you need one device group per RDF group (RA group), even if you have only one frame on that side.

The following examples are based on the 2 by 2 configuration shown in Figure 6-8 “Devices and Symmetrix Units in M by N Configurations”.

Create device groups using the following commands on each node on the R1 side:

# symdg -type RDF1 create dgoraA

# symdg -type RDF1 create dgoraB

For each node on the R2 side (node3 and node4), create the device groups as follows. Note that you have to create two device groups because device groups do not span frames. Do the following on each node on the R2 side:

# symdg -type RDF2 create dgoraA

# symdg -type RDF2 create dgoraB

For each node on the R1 side (node1 and node2), assign the R1 devices to the device groups as follows:

# symld -sid 638 -g dgoraA add dev 00C

# symld -sid 638 -g dgoraA add dev 00D

# symld -sid 130 -g dgoraB add dev 010

# symld -sid 130 -g dgoraB add dev 011

For each node on the R2 side (node3 and node4), assign the R2 devices to the device groups as follows:

# symld -sid 021 -g dgoraA add dev 018

# symld -sid 021 -g dgoraA add dev 019

# symld -sid 363 -g dgoraB add dev 050

# symld -sid 363 -g dgoraB add dev 051

On each node on the R2 side (node3 and node4), associate the local BCV devices to the R2 device group:

# symbcv -g dgoraA add dev 01A

# symbcv -g dgoraA add dev 01B

# symbcv -d dgoraB add dev 052

# symbcv -d dgoraB add dev 053

To manage the BCV devices from the R1 side, you need to associate the BCV devices with the device groups that are configured on the R1 side. Use the following commands on hosts directly connected to the R1 Symmetrix:

# symbcv -g dgoraA associate dev 01A -rdf

# symbcv -g dgoraA associate dev 01B -rdf

# symbcv -g dgoraB associate dev 052 -rdf

# symbcv -g dgoraB associate dev 053 -rdf

Now establish the BCV devices using the following commands from the R2 side:

# symmir -g dgoraA -full est

# symmir -g dgoraB -full est

Alternatively, you can establish the BCV devices with the following commands from the R1 side:

# symmir -g dgoraA -full est -rdf

# symmir -g dgoraB -full est -rdf

Configuring Gatekeeper Devices

You need a gatekeeper device for each device group in the consistency group that will be built in a later step. Use the following commands on all nodes on the R1 side to define gatekeepers and associate them with device groups:

# symgate -sid 638 define dev 010

# symgate -sid 130 define dev 009

# symgate -sid 638 -g dgoraA associate dev 010

# symgate -sid 130 -g dgoraB associate dev 009

Use the following commands on all nodes on the R2 side to define gatekeepers and associate them with device groups:

# symgate -sid 021 define dev 002

# symgate -sid 363 define dev 00B

# symgate -sid 021 -g dgoraA associate dev 002

# symgate -sid 363 -g dgoraB associate dev 00B

Defining the Consistency Groups

To configure consistency groups for use with Metrocluster/SRDF, you first create device groups and gatekeeper groups as described in previous sections. Then you create the consistency groups and add the devices to them as well.

NOTE: Each package requires its own unique consistency group name.

Use the following steps for each package:

  1. On each node in the cluster, create an empty consistency group using the symcg command:

    # symcg create cgoradb

    You can use the same name on all nodes.

  2. Add each device that is going to be used in the consistency group. Use the appropriate SID numbers and device names for the data center that the node is a part of. For example, on node 1 and node 2 in Data Center A:

    # symcg -cg cgoradb -sid 638 add dev 00C

    # symcg -cg cgoradb -sid 638 add dev 00D

    # symcg -cg cgoradb -sid 130 add dev 010

    # symcg -cg cgoradb -sid 130 add dev 011

    And on node 3 and node 4 in Data Center B:

    # symcg -cg cgoradb -sid 021 add dev 018

    # symcg -cg cgoradb -sid 021 add dev 019

    # symcg -cg cgoradb -sid 363 add dev 050

    # symcg -cg cgoradb -sid 363 add dev 051

  3. Enable the consistency group:

    NOTE: This important step must be carried out on every node.

    # symcg -g cgoradb enable

  4. Establish the BCV devices in the secondary Symmetrix as a mirror of the standard device. From either node3 or node4, issue the following commands:

    # symmir -g dgoraA -full est

    # symmir -g dgoraB -full est

    Alternatively, from either node1 or node2, issue the following commands:

    # symmir -g dgoraA -full est -rdf

    # symmir -g dgoraB -full est -rdf

Creating Volume Groups

The following procedures assume you are creating volume groups for a cluster with a 2 by 2 Symmetrix configuration like that of Figure 6-8 “Devices and Symmetrix Units in M by N Configurations”. Use the following steps on node1:

  1. Create the physical volumes:

    # pvcreate -f /dev/rdsk/c6t0d0

    # pvcreate -f /dev/rdsk/c6t0d1

    # pvcreate -f /dev/rdsk/c5t0d2

    # pvcreate -f /dev/rdsk/c5t0d3

  2. Create the directories and special files for the volume groups.

    # mkdir /dev/vgoraA

    # mkdir /dev/vgoraB

    # mknod /dev/vgoraA/group c 64 0x01000

    # mknod /dev/vgoraB/group c 64 0x02000

  3. Create the volume groups. Be careful not to span Symmetrix frames:

    # vgcreate /dev/vgoraA /dev/rdsk/c6t0d0

    # vgextend /dev/vgoraA /dev/rdsk/c6t0d1

    # vgcreate /dev/vgoraB /dev/rdsk/c5t0d2

    # vgextend /dev/vgoraB /dev/rdsk/c5t0d3

  4. Create the logical volumes. (XXXX indicates size in MB)

    # lvcreate -L XXXX /dev/vgoraA

    # lvcreate -L XXXX /dev/vgoraB

  5. Install a VxFS file system on the logical volumes:

    # newfs -F vxfs /dev/vgoraA/rlvol1

    # newfs -F vxfs /dev/vgoraB/rlvol1

  6. Create map files to permit exporting the volume groups to other systems:

    # vgchange -a n vgoraA

    # vgchange -a n vgoraB

    # vgexport -v -s -p -m /tmp/vgoraA.map vgoraA

    # vgexport -v -s -p -m /tmp/vgoraB.map vgoraB

  7. Copy the map files to the other nodes in the cluster:

    # rcp /tmp/vgoraA.map node2:/tmp/vgoraA.map

    # rcp /tmp/vgoraB.map node2:/tmp/vgoraB.map

  8. Split the SRDF logical links:

    # symrdf -g dgoraA split -v

    # symrdf -g dgoraB split -v

On node2, node3, and node4, perform the following steps:

  1. Create the volume group directories and special files:

    # mkdir /dev/vgoraA

    # mkdir /dev/vgoraB

  2. Import the volume groups to each system:

    # vgimport -v -s -m /tmp/vgoraA.map vgoraA

    # vgimport -v -s -m /tmp/vgoraB.map vgoraB

  3. After importing volume groups to all the other nodes, establish SRDF links:

    # symrdf -gdgoraA establish -v

    # symrdf -gdgoraB establish -v

Creating VxVM Disk Groups for Use with Metrocluster/SRDF

If you are using VERITAS storage, use the following procedure to create disk groups. It is assumed that you have already created a VERITAS root disk (rootdg) on the system where you are configuring the storage. The following section shows how to set up VERITAS disk groups. On one node do the following:

  1. Check to make sure the devices are in a synchronized state:

    # symrdf -g dgoraA query

    # symrdf -g dgoraB query

  2. Initialize disks to be used with VxVM by running the vxdisksetup command.

    # vxdisksetup -i c5t0d0

  3. Create the disk group to be used by using the vxdg command. On the primary system run the following command:

    # vxdg init logdata /dev/dsk/c5t0d2 /dev/dsk/c5t0d3 /dev/dsk/c5t0d0 /dev/dsk/c5t0d1

  4. Verify the configuration with the following command:

    # vxdg list

  5. Use the vxassist command to create the logical volume.

    # vxassist -g logdata make logfile 2048m

  6. Verify the configuration with the following command:

    # vxprint -g logdata

  7. Make the filesystem with the following command:

    # newfs -F vxfs /dev/vx/rdsk/logdata/logfile

  8. Create a directory to mount the volume group.

    # mkdir /logs

  9. Mount the volume group.

    # mount /dev/vx/dsk/logdata/logfile /logs

  10. Check if file system exits, then unmount the file system

    # umount /logs

Validating VxVM Disk Groups for Use with Metrocluster/SRDF

The following section shows how to validate VERITAS diskgroups. On one node do the following:

  1. Deport the disk group.

    # vxdg deport logdata

  2. Enable other cluster nodes to have access to the disk group.

    # vxdctl enable

  3. Split the SRDF link to enable R2 Read/Write permission.

    # symrdf -g dgoraA split

    # symrdf -g dgoraB split

  4. Import the disk group.

    # vxdg -tfC import logdata

  5. Start the logical volume in the disk group.

    # vxvol -g logdata startall

  6. Create a directory to mount the volume.

    # mkdir /logs

  7. Mount the volume

    # mount /dev/vx/dsk/logdata/logfile /logs

  8. Check to make sure the file system is present, then unmount the file system.

    # umount /logs

  9. Establish the SRDF link:

    # symrdf -g devgrpA establish

Configuring Serviceguard Packages for Automatic Disaster Recovery

Before you can implement these procedures you must:

  • Configure your cluster hardware according to disaster tolerant architecture guidelines. See Chapter 3 “Designing a Metropolitan Cluster”.

  • Configure the Serviceguard cluster according to the procedures outlined in Managing Serviceguard manual.

  • Create the SymCLI database, and build Symmetrix device groups, consistency groups, and gatekeepers for each package. Export exclusive volume groups for each package as described in “Preparing the Cluster for Data Replication”. This must be done on each node that will potentially run the package.

  • Install the Metrocluster with EMC SRDF product on all nodes according to the instructions in the Metrocluster with EMC SRDF Release Notes.

When you have completed these steps, packages will be able to automatically fail over to an alternate node in another data center and still have access to the data it needs to function.

If either of the operations, failover or R1/R2 swapping, fails to perform, the package will not be automatically started. If the failover operation succeeds, but R1/R2 swapping fails, the user may need to try the swap operation if desired and start the package manually.

This procedure must be repeated on all the cluster nodes for each Serviceguard application package so the application can fail over to any of the nodes in the cluster. Customizations include setting environment variables and supplying customer-defined run and halt commands, as appropriate. The package control script must also be customized for the particular application software that it will control. Consult Managing Serviceguard for more detailed instructions on how to start, halt, and move packages and their services between nodes in a cluster.

For ease of troubleshooting, you may want to configure and test one package at a time.

  1. Create a directory /etc/cmcluster/pkgname for each package:

    # mkdir /etc/cmcluster/pkgname

  2. Create a package configuration file with the commands:

    # cd /etc/cmcluster/pkgname

    # cmmakepkg -p pkgname.ascii

    Customize the package configuration file as appropriate to your application. Be sure to include the pathname of the control script (/etc/cmcluster/pkgname/pkgname.cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters.

  3. In the <pkgname>.ascii file, list the node names in the order in which you want the package to fail over. It is recommended for performance reasons, that you have the package fail over locally first, then to the remote data center.

    NOTE: If you are using the EMS disk monitor as a package resource, you must not use NO_TIMEOUT. Otherwise, package shutdown will hang if there is not access from the host to the package disks.

    This toolkit may increase package startup time by 5 minutes or more. Packages with many disk devices will take longer to start up than those with fewer devices due to the time needed to get device status from the EMC Symmetrix disk array. Clusters with multiple packages that use devices on the EMC Symmetrix disk array will cause package startup time to increase when more than one package is starting at the same time.

    The value of RUN_SCRIPT_TIMEOUT in the package ASCII file should be set to NO_TIMEOUT or to a large enough value to take into consideration the extra startup time due to getting status from the Symmetrix.

  4. Create a package control script with the command:

    # cmmakepkg -s pkgname.cntl

    Customize the control script as appropriate to your application using the guidelines in Managing Serviceguard. Standard Serviceguard package customizations include modifying the VG, LV, FS, IP, SUBNET, SERVICE_NAME, SERVICE_CMD and SERVICE_RESTART parameters. Be sure to set LV_UMOUNT_COUNT to 1 or greater.

  5. Add customer-defined run and halt commands in the appropriate places according to the needs of the application. See Managing Serviceguard for more information on these functions.

  6. In the package_name.ascii file, list the node names in the order in which you want the package to fail over. It is recommended for performance reasons, that you have the package fail over locally first, then to the remote data center. Be sure to set the MAX_CONFIGURED_PACKAGES parameter to 1 or more, depending on the number of packages that will run on the cluster.

  7. Copy the environment file template /opt/cmcluster/toolkit/SGSRDF/srdf.env to the package directory, naming it pkgname_srdf.env:

    # cp /opt/cmcluster/toolkit/SGSRDF/srdf.env \

      /etc/cmcluster/pkgname/pkgname_srdf.env

  8. Edit the environment file <pkgname>_srdf.env as follows:

    1. Add the path where the SymCLI software binaries have been installed to the PATH environment variable. If the software is in the usual location, /usr/symcli/bin, you can just uncomment the line in the script.

    2. Uncomment AUTO*environment variables. It is recommended that you retain the default values of these variables unless you have a specific business requirement to change them. See Appendix B for an explanation of these variables.

    3. Uncomment the PKGDIR variable and set it to the full path name of the directory where the control script has been placed. This directory must be unique for each package and is used for status data files. For example, set PKGDIR to /etc/cmcluster/package_name, removing any quotes around the file names.

    4. Uncomment the DEVICE_GROUP variables EMC Symmetrix for the local disk array and set it to the Symmetrix device group names given in the symdg list command. If you are using an M by N configuration, configure the DEVICE_GROUP variable with the name of the consistency group.

    5. Uncomment the RETRY and RETRYTIME variables. These variables are used to decide how often and how many times to retry the Symmetrix status commands. The defaults should be used for the first package. For other packages RETRYTIME should be altered to avoid contention when more than one package is starting on a node. RETRY * RETRYTIME should be approximately five minutes to keep package startup time under 5 minutes.

       RETRYTIMERETRY
      pkgA60 seconds5 attempts
      pkgB43 seconds7 attempts
      pkgC33 seconds9 attempts

    6. Uncomment the CLUSTERTYPE variable and set it to METRO. (The value CONTINENTAL is only for use with the Continentalclusters product, described in Chapter 5.)

    7. If you are using an M by N configuration, be sure that the variable CONSISTENCYGROUPS is set to 1 in the control script:

      CONSISTENCYGROUPS=1

  9. Distribute Metrocluster/SRDF configuration, environment and control script files to other nodes in the cluster by using ftp or rcp:

    # rcp -p /etc/cmcluster/pkgname/* \

    other_node:/etc/cmcluster/pkgname

    See the example script Samples/ftpit to see how to semi-automate the copy using ftp. This script assumes the package directories already exist on all nodes.

    Using ftp may be preferable at your organization, since it does not require the use of a.rhosts file for root. Root access via .rhosts may create a security issue.

  10. Verify that each node in the Serviceguard cluster has the following files in the directory /etc/cmcluster/pkgname:

    pkgname.cntl

    ServiceGuard package control script

    pkgname_srdf.env

    Metrocluster/SRDF environment file

    pkgname.ascii

    ServiceGuard package ASCII configuration file

    pkgname.sh

    Package monitor shell script, if applicable

    other files

    Any other scripts you use to manage Serviceguard packages

    The Serviceguard cluster is ready to automatically switch packages to nodes in remote data centers using Metroclusters/SRDF

  11. Check the configuration using the cmcheckconf -P package_name.ascii, then apply the Serviceguard configuration using the cmapplyconf -P package_name.ascii command or SAM.

  12. Restore the SRDF logical links for the disks associated with the application package. See the script Samples/post.cmapply for an example of how to automate this task. The script must be customized with the Symmetrix device group names. You may want to redirect the output of this script to a file for debugging purposes.

Maintaining a Cluster that Uses Metrocluster/SRDF

While the cluster is running, all EMC Symmetrix disk arrays that belong to the same Serviceguard package, and are defined in a single SRDF group must be in the same state at the same time. Manual changes of these states can cause the package to halt due to unexpected conditions. In general, it is recommended that no manual change of states be performed while the package and the cluster are running.

There might be situations when the package has to be taken down for maintenance purposes without having the package move to another node.

The following procedure is recommended for normal maintenance of the Metrocluster with EMC SRDF:

  1. Stop the package with the appropriate Serviceguard command.

    # cmhaltpkg pkgname

  2. Split the logical SRDF links for the package.

    # Samples/pre.cmquery

  3. Distribute the Metrocluster with EMC SRDF configuration changes.

    # cmapplyconf -P pkgconfig

  4. Restore the logical SRDF links for the package.

    # Samples/post.cmapply

  5. Start the package with the appropriate Serviceguard command:

    # cmmodpkg -e pkgname

No checking of the status of the SA/FA ports is done. It is assumed that at least one PVLink is functional. Otherwise, the VG activation will fail.

Planned maintenance is treated the same as a failure by the cluster. If you take a node down for maintenance, package failover and quorum calculation is based on the remaining nodes. Make sure that nodes are taken down evenly at each site, and that enough nodes remain on-line to form a quorum if a failure occurs. See “Example Failover Scenarios with Two Arbitrators”.

Managing Business Continuity Volumes

The use of Business Continuity Volumes is recommended with all implementations of Metrocluster/SRDF, and it is required with M by N configurations, which employ consistency groups. These BCV devices will provide a good copy of the data when it is necessary to recover from a rolling disaster—a second failure that occurs while we are attempting to recover from the first failure.

Protecting against Rolling Disasters

An example of a rolling disaster with Metrocluster/SRDF is as follows. At time T0, all the SRDF links go down. The application continues to run on the R1 side. At time T1, the SRDF links are restored, and at T2 a manual resynchronization is started to resync new data from the R1 to the R2 side. At time T3, while resynchronization is in progress, the R1 site fails, and the application starts up on the R2 side. Since the resynchronization did not complete when there was a failure on the R1 side, the data on the R2 side is corrupt.

Using the BCV in Resynchronization

In general, you use the business continuity volumes only in cases where a resynchronization will take place. First, you split off a consistent copy of the data, then perform the resynchronization, then re-establish the BCV mirroring. To perform these steps, use the following sequence of commands:

  1. Split the BCV in the secondary Symmetrix from the mirror group to save a good copy of the data. From node3 or node4:

    # symmir -g dgoraA split

    # symmir -g dgoraB split

    Alternatively, from node1 or node2:# symmir -g dgoraA split -rdf

    # symmir -g dgoraB split -rdf

  2. Resync the data from R1 to R2 devices:

    # symrdf -cg cgoradb est

  3. Use the following command to monitor the status of the RDF pair state between the R1 and R2 devices:

    # symrdf -cg cgoradb query

    Once the column with the “RDF Pair STATE” shows the state as “Synchronized” for all the devices, you can proceed to the next step.

  4. Re-establish the BCV devices in the secondary Symmetrix as a mirror of the standard device. From node 3 or node4:

    # symmir -g dgoraA -full est

    # symmir -g dgoraB -full est

    Alternatively, from node1 or node2:

    # symmir -g dgoraA -full est -rdf

    # symmir -g dgoraB -full est -rdf

R1/R2 Swapping

This section shows how the R1/R2 swapping can be done via Metrocluster/SRDF package and manual procedures. Each of these allow you to swap the SRDF personality for each device designations of a specified devicegroup, where each source R1 device(s) becomes a target R2 device(s) and a target R1 device(s) becomes a source R1 device(s).

R1/R2 Swapping Using MC/SRDF Packages

The Metrocluster/SRDF package can be configured to automatically do R1/R2 swapping upon package failover. To enable R1/R2 swapping in the package, set the environment variable AUTOSWAPR2 in the <package_name>_srdf.env file to 1. Since the swap is done automatically upon package start up, the Metrocluster/SRDF software will only do the swap if the Symmetrix frames and the SRDF links between them are working properly, that is, the SRDF state of the device group is in Synchronized state. If the failover and swap operations are successful, the devices will have their personalities switched, and the data replication will continue from the new R1 devices to the new R2 devices.

If failover or R1/R2 swapping fails, the package will not be automatically started. However, if the failover operation succeeds, but R1/R2 swapping fails, you will need to start the package manually with doing the swap operation. If desired, you can do the swap operation at a later time.

Note that when failing over a package with R1/R2 swapping, the package startup time will be longer than without swapping.

R1/R2 Swapping Using Manual Procedures

You can also do R1/R2 swapping manually. There are two scenarios where manual swapping is supported by Metrocluster/SRDF.

Scenario 1: In this scenario, the package failover is due to host failure or due to planned downtime maintenance. The SRDF links and the Symmetrix frames are still up and running. Because the package startup time will be longer if the swapping is done automatically, the user can choose not to have the swapping done by the package and then manually execute the swapping after the package is up and running on the R2 side. Following is the manual procedure to swap the devices personalities and change the direction of the data replication.

On the host that connects to the R2 side use the following steps:

1. Swap the personalities of the devices and mark the old R1 devices to be refresh from the old R2 devices.

# symrdf -g <device_group> swap -refresh R1

2. After swapping is completed, the devices will be in Suspended state. Next establish the device group for data replication from the new R1 devices to the new R2 devices.

# symrdf -g <device_group> establish

Scenario 2: In this scenario, two failures happen before the package fails over to the secondary data center. The SRDF link fails; the package continues to run and write data on R1 devices. Sometime later, the host fails; the package then fails over to the secondary data center. In this case, even if the AUTOSWAPR2 variable is set to 1, the package will not do the R1/R2 swapping, which happens after the host in the primary data center and the SRDF links are fixed. To minimize the application down time, instead of failing the application back to the primary data center, you can leave the application running in the secondary data center, and then manually swap the devices personalities and change the direction of the data replication.

1. Swap the personalities of the devices and mark the old R1 devices to be refresh from the old R2 devices.

# symrdf -g <device_group> swap -refresh R1

2. After swapping is completed, the devices will be in suspended state. Next Establish the device groups for data replication from the new R1 devices to the new R2 devices.

# symrdf -g <device_group> establish

WARNING! R1/R2 Swapping cannot be used in an M by N Configuration.


Preparing the Cluster for Data Replication
  		 


Building a Continental Cluster Solution with EMC SRDF  
Printable version
Privacy statement Using this site means you accept its terms Feedback to webmaster
© Hewlett-Packard Development Company, L.P.