 |
» |
|
|
|
Setting
up 1 by 1 Configurations | |
The most common Symmetrix configuration used with Metrocluster
with EMC SRDF is a 1 by 1 configuration in which there is a single Symmetrix
frame at each Data Center. This section describes how to set up
this configuration using EMC Solutions Enabler and HP-UX commands.
It is assumed the Symmetrix CLI database is already set up on each
node, as described in the previous section ““Preparing
the Cluster for Data Replication”.” A basic 1 by 1 configuration is shown in Figure 5-7 “Mapping
HP-UX Device File Names to Symmetrix Units”, which is a graphical view of the data in Table 5-2 “Mapping for a 4 Node Cluster connected to 2 Symmetrix Arrays”.
Creating
Symmetrix Device GroupsA single Symmetrix
device group must be defined for each package on each node that
is connected to the Symmetrix. The following procedure must be done
on each node that may potentially run the package: | | | |  | NOTE: The sample scripts mk3symgrps.nodename can be modified to automate these steps. | | | | |
Use the symdg command, or modify the mk3symgrps.nodename script to define an R1 and an R2 device group for each
package. # symdg create -type RDF1 devgroupname Issue the above command on nodes attached to the R1 side. # symdg create -type RDF2 devgroupname Issue the above command on nodes attached to the R2 side. The group name must be the same on each node on the R1 and
R2 side. The devgroup name used will be later placed in variable DEVICE_GROUP defined in pkg.env file. Use the symld command to add all LUNs that comprise the Volume Group
for that package on that host. The HP-UX device file names for all
Volume Groups that belong to the package must be
defined in one Symmetrix device group. All devices belonging to
Volume Groups that are owned by an application package must be added
to a single Symmetrix device group. # symld -g devgroupname add dev devnumber1 # symld -g devgroupname add dev devnumber2 At this point, it will be helpful to refer to Table 5-2 “Mapping for a 4 Node Cluster connected to 2 Symmetrix Arrays”. Although, the HP-UX device
file names on each node specified may be different, the device group
must be the same on each node. When creating the Symmetrix device groups, specify only one
HP-UX path to a particular Symmetrix device. Do not specify alternate
paths (PVLinks). The EMC Solutions Enabler uses the HP-UX path only
to determine to which Symmetrix device you are referring. The Symmetrix
device may be added to the device group only once. | | | | | NOTE: Symmetrix
Logical Device names must be the default names
of the form DEVnnn (for example, DEV001). Do not use this option for creating your own
device names. | | | | |
The script must be customized for each system including: Particular HP-UX device file names. Symmetrix device group name (an arbitrary, but unique
name may be chosen for each group that defines all of the volume
groups (VGs), which belong to a particular Serviceguard package).
Configuring
Gatekeeper DevicesGatekeeper devices
must be unique per Serviceguard package to prevent contention in
the Symmetrix when commands are issued, such as two or more packages
starting up at the same time. Gatekeeper devices are unique to a
Symmetrix unit. They are not replicated across the SRDF link. Gatekeeper
devices are marked GK in the syminq output, and are usually 2880 KB in size. | | | | | NOTE: The sample scripts mk4gatekpr.nodename can be modified to automate these steps. | | | | |
Define at least two gatekeepers per
package per node (assuming PV links are used). They will only be
available for use by that node. Each gatekeeper device is configured
on different physical links. # symgate -sid sidnumber1 define dev devnumber1 # symgate -sid sidnumber2 define dev devnumber2 Associate the gatekeeper devices with the Symmetrix
device group for that package. # symgate -sid sidnumber1 -g devgroupname \
associate dev devnumber1 # symgate -sid sidnumber2 -g devgroupname \
associate dev devnumber2 Define a pool of four or more additional gatekeeper
devices that are not associated with any particular node. The EMC
Solutions Enabler will switch to an alternate gatekeeper device
if the path to the primary gatekeeper device fails.
Verifying
the EMC Symmetrix ConfigurationWhen
finished with all these steps, use the symrdf list command to get a listing of all devices and their states. Back up the EMC Solutions Enabler database on each node, so
that these configuration steps do not have to be repeated if a failure
corrupts the database. The EMC Solutions Enabler database is a binary
file located in the directory /var/symapi/db. Creating
and Exporting Volume GroupsUse the following procedure to create volume groups and export
them for access by other nodes. The sample script mk1VGs in the /opt/cmcluster/toolkit/SGSRDF/Samples directory can be modified to automate these steps. Define the appropriate
Volume Groups (VGs) on each node that run the application package. # mkdir /dev/vgxx # mknod /dev/vgxx/group c 64 0xnn0000 where the name /dev/vgxx and the number nn are unique within the cluster. Create volume groups only
on the primary system. Use the vgcreate and vgextend commands, specifying the appropriate HP-UX device file
names. # vgcreate vgname /dev/dsk/cxtydz # vgextend vgname /dev/dsk/cxtydz Use the vgchange command to de-activate the volume group and use the
vgexport command with the -p option to export the VGs on the primary system without
removing the HP-UX device files: # vgchange -a n vgname # vgexport -v -s -p -m mapfilename vgname Copy the map files to all of the nodes. The sample script Samples/ftpit shows a semi-automated way (using ftp) to copy the files. Enter the password interactively.
Importing
Volume Groups on Other NodesUse the following procedure to import volume groups. The sample
script mk2imports can be modified to automate these steps: Import the VGs
on all of the other systems that might run the Serviceguard package
and backup the LVM configuration. Make sure that you
split the logical SRDF
links before importing the VGs, especially if you are importing
the VGs on the R2 side. # symrdf -g devgrpname split -v # vgimport -v -s -m mapfilename vgname Back up the configuration. # vgchange -a y vgname # vgcfgbackup vgname # vgchange -a n vgname # symrdf -g devgrpname establish -v See the sample script Samples/mk2imports.
| | | | | NOTE: Exclusive activation must be used
for all volume groups associated with packages that use the EMC.
The design of Metrocluster with EMC SRDF assumes that only one system
in the cluster will have a VG activated at a time. | | | | |
The examples in the previous sections describe the use of
the vgimport and vgexport commands with the -s option. In addition, the mk1VGs script uses a -s in the vgexport command, and the mk2imports script uses a -s in the vgimport command. Optionally, remove this option from both commands
if using PV links. The -s option to the vgexport command saves the volume group id (VGID) in the map file, but it does not preserve the order
of PV links. To specify the exact order of PV links, do not use
the -s option with vgexport, and in the vgimport command, enter the individual links in the desired order,
as in the following example: # vgimport -v -m mapfilename vgname linkname1 linkname2 Grouping
the Symmetrix Devices at Each Data Center | |
The use of R1/R2 devices in M by N configurations of multiple Symmetrix
frames is enabled by means of consistency groups.
A consistency group is a set of Symmetrix RDF devices that are configured to
act in unison to maintain the integrity of a database. Because Metrocluster
with EMC SRDF works at the device group level, the consistency group
is implemented and managed as a single device group even though
it spans multiple Symmetrix frames. Consistency groups are managed
by the EMC PowerPath product. When PowerPath is installed, the Symmetrix
tracks the I/Os that are written to the devices in the consistency
group. If an I/O cannot be written to a remote Symmetrix due to
a failure of a remote device or an RDF link, the data flow to the
other Symmetrix will be halted in less than one second. Once mirroring
is resumed, any updates will propagate with normal SRDF operation. Figure 5-8 “2
X 2 Node and Data Center Configuration with Consistency Groups” shows when there
is a break in the links between two of the Symmetrix frames, the
use of consistency groups (depicted as dashed oval lines) ensures
that the other two links are also suspended. Creating
Symmetrix Device Groups | |
For each node on the R1 side (node1 and node2), create the device groups as follows. Note: It is necessary
to create two device groups since device groups do not span frames. The following examples are based on the configuration shown
in Figure 5-9 “Devices
and Symmetrix Units in M by N Configurations”. Create device
groups using the following commands on each node on the R1 side. # symdg -type RDF1 create dgoraA # symdg -type RDF1 create dgoraB For each node on the R2 side
(node3 and node4), create the device groups as follows. Note: It is necessary
to create two device groups since device groups do not span frames.
Do the following on each node on the R2 side. # symdg -type RDF2 create dgoraA # symdg -type RDF2 create dgoraB For each node on the R1 side
(node1 and node2), assign the R1 devices to the device groups. # symld -sid 638 -g dgoraA add dev 00C # symld -sid 638 -g dgoraA add dev 00D # symld -sid 130 -g dgoraB add dev 010 # symld -sid 130 -g dgoraB add dev 011 For each node on
the R2 side (node3 and node4), assign the R2 devices to the device groups. # symld -sid 021 -g dgoraA add dev 018 # symld -sid 021 -g dgoraA add dev 019 # symld -sid 363 -g dgoraB add dev 050 # symld -sid 363 -g dgoraB add dev 051 On each node on the R2 side
(node3 and node4), associate the local BCV devices to the R2 device group. # symbcv -g dgoraA add dev 01A # symbcv -g dgoraA add dev 01B # symbcv -d dgoraB add dev 052 # symbcv -d dgoraB add dev 053 To manage the BCV devices
from the R1 side, it is necessary to associate the BCV devices with
the device groups that are configured on the R1 side. Use the following
commands on hosts directly connected to the R1 Symmetrix. # symbcv -g dgoraA associate dev 01A -rdf # symbcv -g dgoraA associate dev 01B -rdf # symbcv -g dgoraB associate dev 052 -rdf # symbcv -g dgoraB associate dev 053 -rdf Establish the BCV devices
using the following commands from the R2 side. # symmir -g dgoraA -full est # symmir -g dgoraB -full est Alternatively, establish
the BCV devices with the following commands from the R1 side. # symmir -g dgoraA -full est -rdf # symmir -g dgoraB -full est -rdf
Configuring
Gatekeeper Devices | |
It is necessary to have a gatekeeper device for each device
group in the consistency group that will be built in a later step.
Use the following commands on all nodes on the R1 side to define
gatekeepers and associate them with device groups. # symgate -sid 638 define dev 010 # symgate -sid 130 define dev 009 # symgate -sid 638 -g dgoraA associate dev 010 # symgate -sid 130 -g dgoraB associate dev 009 Use the following commands on all nodes on the R2 side to
define gatekeepers and associate them with device groups. # symgate -sid 021 define dev 002 # symgate -sid 363 define dev 00B # symgate -sid 021 -g dgoraA associate dev 002 # symgate -sid 363 -g dgoraB associate dev 00B Creating
the Consistency Groups | |
To configure consistency groups for using Metrocluster with
EMC SRDF, first create device groups and gatekeeper groups as described
in previous sections. The following examples are based on the configuration shown
in Figure 5-9 “Devices
and Symmetrix Units in M by N Configurations”. Use the following steps for each package: On each node in the cluster,
create an empty consistency group using the symcg command. # symcg create cgoradb Use the same name on all nodes. Add each device that is going to be used in the
consistency group. Use the appropriate SID numbers and device names
for the data center that the node is a part of. For example, on node1 and node2 in Data Center A. # symcg -cg cgoradb -sid 638 add dev 00C # symcg -cg cgoradb -sid 638 add dev 00D # symcg -cg cgoradb -sid 130 add dev 010 # symcg -cg cgoradb -sid 130 add dev 011 And on node3 and node4 in Data Center B. # symcg -cg cgoradb -sid 021 add dev 018 # symcg -cg cgoradb -sid 021 add dev 019 # symcg -cg cgoradb -sid 363 add dev 050 # symcg -cg cgoradb -sid 363 add dev 051 Enable the consistency group. # symcg -g cgoradb enable | | | | | NOTE: This important step must be carried out on every node. | | | | |
Establish the BCV devices in the secondary Symmetrix
as a mirror of the standard device. From either node3 or node4. # symmir -cg cgoradb -full est # symmir -cg cgoradb -full est Alternatively, from either node1 or node2. # symmir -cg cgoradb -full est -rdf
Creating
Volume Groups | |
The following procedures assume the volume groups being created
for a cluster and the device groups, as shown in Figure 5-9 “Devices
and Symmetrix Units in M by N Configurations”. Use the following steps on node1: Create the physical volumes. # pvcreate -f /dev/rdsk/c6t0d0 # pvcreate -f /dev/rdsk/c6t0d1 # pvcreate -f /dev/rdsk/c5t0d2 # pvcreate -f /dev/rdsk/c5t0d3 Create the directories and special files for the
volume groups. # mkdir /dev/vgoraA # mkdir /dev/vgoraB # mknod /dev/vgoraA/group c 64 0x01000 # mknod /dev/vgoraB/group c 64 0x02000 Create the volume groups. Be careful not to span
Symmetrix frames. # vgcreate /dev/vgoraA /dev/rdsk/c6t0d0 # vgextend /dev/vgoraA /dev/rdsk/c6t0d1 # vgcreate /dev/vgoraB /dev/rdsk/c5t0d2 # vgextend /dev/vgoraB /dev/rdsk/c5t0d3 Create the logical volumes. (XXXX indicates size
in MB) # lvcreate -L XXXX /dev/vgoraA # lvcreate -L XXXX /dev/vgoraB Install a VxFS file system on the logical volumes. # newfs -F vxfs /dev/vgoraA/rlvol1 # newfs -F vxfs /dev/vgoraB/rlvol1 Create map files to permit exporting the volume
groups to other systems. # vgchange -a n vgoraA # vgchange -a n vgoraB # vgexport -v -s -p -m /tmp/vgoraA.map vgoraA # vgexport -v -s -p -m /tmp/vgoraB.map vgoraB Copy the map files to the other nodes in the cluster. # rcp /tmp/vgoraA.map node2:/tmp/vgoraA.map # rcp /tmp/vgoraB.map node2:/tmp/vgoraB.map Split the SRDF logical links. # symrdf -g dgoraA split -v # symrdf -g dgoraB split -v
On node2, node3, and node4, perform the following steps. Create
the volume group directories and special files. # mkdir /dev/vgoraA # mkdir /dev/vgoraB Import the volume groups to each system: # vgimport -v -s -m /tmp/vgoraA.map vgoraA # vgimport -v -s -m /tmp/vgoraB.map vgoraB After importing volume groups to all the other nodes,
establish SRDF links. # symrdf -gdgoraA establish -v # symrdf -gdgoraB establish -v
Creating
VxVM Disk Groups using Metrocluster with EMC SRDF | |
If using VERITAS storage, use the following procedure to create
disk groups. It is assumed VERITAS root disk (rootdg) has been created on the system where configuring the
storage. The following section shows how to set up VERITAS disk
groups. On one node do the following: Check to make sure the devices are
in a synchronized state. # symrdf -g dgoraA query # symrdf -g dgoraB query Initialize disks to be used with VxVM by running
the vxdisksetup command. # vxdisksetup -i c5t0d0 Create the disk group to be used by using the vxdg command on the primary system. # vxdg init logdata /dev/dsk/c5t0d2 /dev/dsk/c5t0d3 \ /dev/dsk/c5t0d0 /dev/dsk/c5t0d1 Verify the configuration. # vxdg list Create the logical volume. # vxassist -g logdata make logfile 2048m Verify the configuration. # vxprint -g logdata Make the filesystem. # newfs -F vxfs /dev/vx/rdsk/logdata/logfile Create a directory to mount the volume group. # mkdir /logs Mount the volume group: # mount /dev/vx/dsk/logdata/logfile /logs Check if file system exits, then unmount the file
system. # umount /logs
Validating
VxVM Disk Groups using Metrocluster with EMC SRDF | |
The following section shows how to validate VERITAS diskgroups.
On one node do the following: Deport the disk group. # vxdg deport logdata Enable other cluster nodes to have access to the
disk group. # vxdctl enable Split the SRDF link to enable R2 Read/Write permission. # symrdf -g dgoraA split # symrdf -g dgoraB split Import the disk group. # vxdg -tfC import logdata Start the logical volume in the disk group. # vxvol -g logdata startall Create a directory to mount the volume. # mkdir /logs Mount the volume. # mount /dev/vx/dsk/logdata/logfile /logs Check to make sure the file system is present, then
unmount the file system. # umount /logs Establish the SRDF link. # symrdf -g devgrpA establish
| | | | | NOTE: In a Metrocluster/SRDF environment, VxVM commands should
not be run against write-disabled disks. This is due to VxVM potentially putting
these disks into an offline state. Subsequent activation of a VxVM
disk group might fail when the disks are again write-enabled, and requires
a vxdisk scandisks to be executed prior to
disk group activation. | | | | |
Additional
Examples of M by N Configurations | |
Figure 5-10 “2
by 1 Configuration” shows a 2 by 1
configuration with BCV’s, which indicates R1 volumes at
Data Center A and R2 volumes and BCVs at Data Center B for pkg A
and pkg B. Figure 5-11 “Bidirectional
2 by 2 Configuration” shows a bidirectional
2 by 2 configuration with additional packages on node3 and node4, and R1 and R2 volumes at both data centers. In this
configuration, R1 volumes and pkg A and pkg B are at Data Center A, and R2 volumes are at Data Center
B. R1 volumes for pkg C and pkg D are at Data Center B, and R2 volumes are at Data Center
A. Configuring
Serviceguard Packages for Automatic Disaster Recovery | |
Before implementing
these procedures it is necessary to do the following: Configure your cluster hardware according
to disaster tolerant architecture guidelines. See the Understanding
and Designing Serviceguard Disaster Tolerant Architectures user’s
guide. Configure the Serviceguard cluster according to
the procedures outlined in Managing Serviceguard user’s
guide. Create the EMC Solutions Enabler database, and build
Symmetrix device groups, consistency groups, and gatekeepers for
each package. Export exclusive volume groups for each package as
described in “Preparing
the Cluster for Data Replication”.
This must be done on each node that will potentially run the package. Install the Metrocluster EMC SRDF product on all
nodes according to the instructions in the Metrocluster
with EMC SRDF Release Notes.
When these steps have been completed, packages will be able
to automatically fail over to an alternate node in another data
center and still have access to the data it needs to function. This procedure must be repeated on all the cluster nodes for
each Serviceguard application package so the application can fail
over to any of the nodes in the cluster. Customizations include
setting environment variables and supplying customer-defined run
and halt commands, as appropriate. The package control script must
also be customized for the particular application software that
it will control. Consult the Managing Serviceguard user’s
guide for more detailed instructions on how to start, halt, and
move packages and their services between nodes in a cluster. For ease of troubleshooting, it is recommended to configure
and test one package at a time. Create a directory
/etc/cmcluster/pkgname for each package. # mkdir /etc/cmcluster/pkgname Create a package configuration
file. # cd /etc/cmcluster/pkgname # cmmakepkg -p pkgname.ascii Customize the package configuration file as appropriate to
your application. Be sure to include the pathname of the control
script (/etc/cmcluster/pkgname/pkgname.cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters. In the <pkgname>.ascii file, list the node names in the order for which the
package is to fail over. It is recommended for performance reasons,
that the package fail over locally first, then to the remote data
center. | | | | | NOTE: If using the EMS disk monitor as a package resource,
do not use NO_TIMEOUT. Otherwise, package shutdown will hang if there
is not access from the host to the package disks. | | | | |
This toolkit may increase package startup time by 5 minutes
or more. Packages with many disk devices will take longer to start
up than those with fewer devices due to the time needed to get device status
from the EMC Symmetrix disk array. Clusters with multiple packages
that use devices on the EMC Symmetrix disk array will cause package
startup time to increase when more than one package is starting
at the same time. The value of RUN_SCRIPT_TIMEOUT in
the package ASCII file should be set to NO_TIMEOUT or
to a large enough value to take into consideration the extra startup
time due to getting status from the Symmetrix. Create a package control
script. # cmmakepkg -s pkgname.cntl Customize the control script as appropriate to your application
using the guidelines in the Managing Serviceguard user’s
guide. Standard Serviceguard package customizations
include modifying the VG, LV, FS, IP, SUBNET, SERVICE_NAME, SERVICE_CMD,
and SERVICE_RESTART parameters. Be sure
to set LV_UMOUNT_COUNT to 1 or greater. Add customer-defined run and halt commands in the
appropriate places according to the needs of the application. See
the Managing Serviceguard user’s
guide for more information on these functions. In the package_name.ascii file, list the node names in the order in which you
want the package to fail over. It is recommended, for performance
reasons, that the package fail over locally first, then to the remote
data center. For the MAX_CONFIGURED_PACKAGES parameter, the minimum value is 0 and maximum
default value is 150 (depending on the number of packages that will
run on the cluster). Copy the environment file template /opt/cmcluster/toolkit/SGSRDF/srdf.env to the package directory, naming it pkgname_srdf.env: # cp /opt/cmcluster/toolkit/SGSRDF/srdf.env \ /etc/cmcluster/pkgname/pkgname_srdf.env | | | | | NOTE: If not use a package name as a filename for the package
control script, it is necessary to follow the convention of the
environment file name. This is the combination of the file name
of the package control script without the file extension, an underscore
and type of the data replication technology (srdf) used. The extension .env of the file must be used. The following examples demonstrate
how the environment file name should be chosen:
Example
1: If the file name of the control script is pkg.cntl, the environment file name would be pkg_srdf.env.
Example 2: If the file name of
the control script is control_script.sh, the environment file name would be control_script_srdf.env. | | | | |
Edit the environment file as follows: Add the path where the EMC Solutions
Enabler software binaries have been installed to the PATH environment
variable. If the software is installed in the default location, /usr/symcli/bin, there is no need to set the PATH environment variable
in this file. Uncomment AUTO*environment
variables. It is recommended to retain the default values of these
variables unless there is a specific business requirement to change
them. See Appendix B for an explanation of these variables. Uncomment the PKGDIR variable
and set it to the full path name of the directory where the control
script has been placed. This directory must be unique for each package
and is used for status data files. For example, set PKGDIR to /etc/cmcluster/package_name, removing any quotes around the file names. Uncomment the RDF_MODE variables and set it to the RDF mode for RDF pairs in
the device group to be Synchronous (sync) or Asynchronous (async). Uncomment the DEVICE_GROUP variables EMC Symmetrix for the local disk array
and set it to the Symmetrix device group names given in the symdg list command. If you are using an M by N configuration, configure
the DEVICE_GROUP variable with the name of the consistency group. Uncomment the RETRY and RETRYTIME variables.
These variables are used to decide how often and how many times
to retry the Symmetrix status commands. The defaults should be used
for the first package. For other packages RETRYTIME should
be altered to avoid contention when more than one package is starting
on a node. RETRY * RETRYTIME should
be approximately five minutes to keep package startup time under
5 minutes. | | RETRYTIME | RETRY |
|---|
| pkgA | 60 seconds | 5 attempts | | pkgB | 43 seconds | 7 attempts | | pkgC | 33 seconds | 9 attempts |
Uncomment the CLUSTERTYPE variable
and set it to METRO. (The value CONTINENTAL is only for use with the Continentalclusters product,
described in Chapter 5.) If using an M by N configuration, be sure that the
variable CONSISTENCYGROUPS is set to 1 in the control script CONSISTENCYGROUPS=1
Distribute Metrocluster with
EMC SRDF configuration, environment and control script files to
other nodes in the cluster by using ftp or rcp: # rcp -p /etc/cmcluster/pkgname/* \ other_node:/etc/cmcluster/pkgname See the example script Samples/ftpit to see how to semi-automate the copy using ftp. This script assumes the package directories already
exist on all nodes. Using ftp may be preferable at your organization, since it does
not require the use of a.rhosts file for root. Root access via .rhosts may create a security issue. Verify that each node in the Serviceguard cluster
has the following files in the directory /etc/cmcluster/pkgname: - pkgname.cntl
Serviceguard
package control script - pkgname_srdf.env
Metrocluster EMC
SRDF environment file - pkgname.ascii
Serviceguard
package ASCII configuration file - pkgname.sh
Package
monitor shell script, if applicable - other files
Any
other scripts you use to manage Serviceguard
packages
The Serviceguard cluster is ready to automatically switch
packages to nodes in remote data centers using Metrocluster/SRDF Check the configuration using the cmcheckconf -P package_name.ascii, then apply the Serviceguard configuration using the cmapplyconf -P package_name.ascii command or SAM. Restore
the SRDF logical links for the disks associated with the application
package. See the script Samples/post.cmapply for an example of how to automate this task. The script
must be customized with the Symmetrix device group names. Redirect
the output of this script to a file for debugging purposes.
Maintaining
a Cluster that uses Metrocluster with EMC SRDF | |
While the cluster is running, all EMC Symmetrix disk arrays
that belong to the same Serviceguard package, and are defined in
a single SRDF group must be in the same state
at the same time. Manual changes of these states can cause the package
to halt due to unexpected conditions. In general, it is recommended
that no manual change of states should be performed
while the package and the cluster are running. There might be situations when the package has to be taken
down for maintenance purposes without having the package move to
another node. The following procedure is recommended for normal maintenance
of Metrocluster EMC SRDF: Stop the package with the appropriate
Serviceguard command. # cmhaltpkg pkgname Split the logical SRDF links for the package. # Samples/pre.cmquery Distribute the Metrocluster EMC SRDF configuration
changes. # cmapplyconf -P pkgconfig Restore the logical SRDF links for the package. # Samples/post.cmapply Start the package with the appropriate Serviceguard
command. # cmmodpkg -e pkgname
No checking of the status of the SA/FA ports is done. It is
assumed that at least one PVLink is functional. Otherwise, the Volume
Group activation will fail. Planned maintenance is treated the same as a failure by the
cluster. If the node is taken down for maintenance, package failover
and quorum calculation is based on the remaining nodes. Make sure
that the nodes are taken down evenly at each site, and enough nodes
remain on-line to form a quorum if a failure occurs. For examples
of failover scenarios, see section, “Example
Failover Scenarios with Two Arbitrators”. Managing
Business Continuity Volumes | |
The use of Business Continuity Volumes is recommended with
all implementations of Metrocluster EMC SRDF, and it is required with
M by N configurations, which employ consistency groups. These BCV devices
will provide a good copy of the data when it is necessary to recover
from a rolling disaster—a second
failure that occurs while attempting to recover from the first failure. Protecting
against Rolling DisastersThe following is an example of a rolling disaster with Metrocluster
with EMC SRDF At time T0, all the SRDF links go down. The application continues
to run on the R1 side. At time T1, the SRDF links are restored,
and at T2 a manual resynchronization is started to resync new data
from the R1 to the R2 side. At time T3, while resynchronization
is in progress, the R1 site fails, and the application starts up
on the R2 side. Since the resynchronization did not complete when
there was a failure on the R1 side, the data on the R2 side is corrupt. Using
the BCV in ResynchronizationIn the case described above, you use the business continuity
volumes, which protect against a rolling disaster. First split off
a consistent copy of the data at the recovery site, and then perform
the re-synchronization. After the re-synchronization is complete,
re-establish the BCV mirroring. To protect data consistency on R2
in rolling disaster, use the following procedures: Before starting the re-synchronization
from R1 to R2 side, it is necessary to disable the package switch
capability to prevent the package automatically fail over to R2
if a new disaster occurs when the re-sync is still in progress.
To disable the package switching on the R2 nodes. # cmmodpkg -d pkgname -n node_name Split the BCV in the secondary Symmetrix from the
mirror group to save a good copy of the data from nodes on R2 side. # symmir -g dgname split Alternatively, from node on R1 side. # symmir -g dgname split -rdf Begin to resynchronize the data from R1 to R2 devices. # symrdf -g dgname est After the resynchronization is completed, enable
the package switching on the node on R2 side. # cmmodpkg -e pkgname -n node_name Re-establish the BCV to
R2 devices on R2 as a mirror. # symmir -g dgname -full est Alternatively, from node on R1 side. # symmir -g dgname -full est -rdf
In Metrocluster with EMC SRDF environment, following the resynchronization
process described above, which prevents the package from automatically
failing over and starting on the R2 side if a disaster takes place
when the resync is in progress. This ensures the package would not
automatically start and operate on the inconsistent data in the event
of a rolling disaster. As demonstrated above, the re-sync is a manual process and
initiated by an operator after the links are fixed. The pairstate of
the devices should be Synchronized for SRDF/Synchronous or Consistent
for SRDF/Asynchronous when the re-sync is completed. Check the state
and ensure that the re-sync is completed before enabling the package
switch. If Metrocluster with EMC SRDF is used in Continentalclusters,
it is not necessary to disable the package switch on the nodes on
recovery site since each site has its own cluster. However, when
the re-sync is in progress, make sure the recovery site will not
start the recovery operation in the event of a disaster occurring
on the primary site. Use the following procedures to protect data
consistency on R2 in a Continentalclusters environment: Split the BCV in the secondary Symmetrix
from the mirror group to save a good copy of the data from nodes
on R2 side: # symmir -g dgname split Alternatively, from node on R1 side. # symmir -g dgname split -rdf Begin to resynchronize the data from R1 to R2 devices. # symrdf -g dgname est Re-establish the BCV to
R2 devices on R2 as a mirror. # symmir -g dgname -full est Alternatively, from node on R1 side. # symmir -g dgname -full est -rdf
R1/R2
Swapping | |
This section describes how the R1/R2 swapping can be done
via the Metrocluster SRDF package and manual procedures. Each of
these methods allows swapping the SRDF personality for each device designation
of a specified device group. In this situation, each source R1 device(s)
becomes a target R2 device(s), and a target R1 device(s) becomes
a source R1 device(s). R1/R2
Swapping using Metrocluster SRDFThe Metrocluster SRDF package can be configured to automatically
do R1/R2 swapping upon package failover. To enable R1/R2 swapping
in the package, set the environment variable AUTOSWAPR2 in the <package_name>_srdf.env file to 1 or 2. Since the swap is done automatically
upon package start up, the Metrocluster SRDF software will only
do the swap if the Symmetrix frames and the SRDF links between them
are working properly, that is, the SRDF state of the device group
is in Synchronized state. If the failover and swap operations succeed,
the devices will have their personalities switched, and the data replication
will continue from the new R1 devices to the new R2 devices. Prior to Metrocluster performing an R1/R2 swap, if the failover
operation fails, the package will not be automatically started.
If the failover operation succeeds, but R1/R2 swapping fails, then
either the package is automatically started or fails depending on
the value of the environment variable AUTOSWAPR2. The environment variable AUTOSWAPR2 can be set to either “1” or “2”. This
will depend on whether the package needs to be started automatically
on R2, in case of R1/R2 swap failure. If AUTOSWAPR2 is set to “1”, the package will fail
to start if R1/R1 swapping fails. In this scenario it is necessary
to start the package manually by doing the swap operation. If preferred,
this can be done at a later time. If AUTOSWAPR2 is set to “2”, the package is automatically
started regardless of a R1/R2 swap failure. In this scenario
the data will not be protected remotely. | | | | | NOTE: When failing over a package with R1/R2 swapping, the
package startup time will be longer than without the swapping. | | | | |
R1/R2
Swapping using Manual ProceduresIt is also possible to do R1/R2 swapping manually. There are
two scenarios where manual swapping is supported by Metrocluster
with EMC SRDF. Scenario 1: In this scenario, the package
failover is due to host failure or due to planned downtime maintenance.
The SRDF links and the Symmetrix frames are still up and running.
Because the package startup time will be longer if the swapping
is done automatically, the user can choose not to have the swapping
done by the package and then manually execute the swapping after
the package is up and running on the R2 side. Following is the manual
procedure to swap the devices personalities and change the direction
of the data replication. On the host that connects to the R2 side, use the following
steps: 1. Swap the personalities of the devices and mark the old
R1 devices to be refresh from the old R2 devices. # symrdf -g <device_group> swap -refresh R1 2. After swapping is completed, the devices will be in Suspended
state. Next establish the device group for data replication from
the new R1 devices to the new R2 devices. # symrdf -g <device_group> establish Scenario 2: In this scenario, two failures
happen before the package fails over to the secondary data center.
The SRDF link fails; the package continues to run and write data
on R1 devices. Sometime later, the host fails; the package then
fails over to the secondary data center. In this case, even if the AUTOSWAPR2 variable is set to 1 or 2, the package will not do
the R1/R2 swapping, which happens after the host in the primary data
center and the SRDF links are fixed. To minimize the application down time, instead of failing
the application back to the primary data center, leave the application
running in the secondary data center. Then manually swap the devices
personalities and change the direction of the data replication. 1. Swap the personalities of the devices and mark the old
R1 devices to be refresh from the old R2 devices. # symrdf -g <device_group> swap -refresh R1 2. After swapping is completed, the devices will be in a suspended
state. Next Establish the device groups for data replication from
the new R1 devices to the new R2 devices. # symrdf -g <device_group> establish | | | |  | CAUTION: R1/R2 Swapping cannot be used in an M
by N Configuration. | | | | |
|
Printable version
