Data Replication Procedures

This section describes the procedures that should be used to manage the data in a Symmetrix cascading failover configuration.

Data Initialization Procedures

The following procedures are needed if you already have existing data prior to implementing this solution. If you already have a metropolitan cluster (that is, the primary Symmetrix and the secondary Symmetrix are up and running, and the SRDF volume pairs between the two Symmetrix frames are already established) and you are now adding the recovery cluster to the configuration, only procedure 2 is required.

1. Mirroring from the Primary to the Secondary Symmetrix

This procedure is illustrated in Figure 8-7 “Mirroring from the Primary to the Secondary Symmetrix”.

Figure 8-7 Mirroring from the Primary to the Secondary Symmetrix

Execute the following commands from a node that connects to the primary Symmetrix:

Incrementally establish the BCV/R1 devices in the secondary Symmetrix as mirrors of the standard devices if they are not already established:
# symmir -g <prisymdevgrpname> est -rdf
Incrementally establish the SRDF volume pairs between the primary Symmetrix and the secondary Symmetrix:# symrdf -g <prisymdevgrpname> est
Start the application to load the data to the R1 devices in the primary Symmetrix.

Refer to the sample script datainit in the /opt/cmcluster/toolkit/SGSRDF/cascade/Samples directory for examples of how these commands are used. This script is designed to run only on a node that connects to the primary Symmetrix.

2. Mirroring from the Secondary to the Recovery Symmetrix

This procedure is illustrated in Figure 8-8 “Mirroring from the Secondary to the Recovery Symmetrix”.

Figure 8-8 Mirroring from the Secondary to the Recovery Symmetrix

Execute the following commands from a node that connects to the primary Symmetrix:

Freeze the application I/O to prevent the application from writing data to the R1 devices in the primary Symmetrix (in case of adding the recovery cluster to the existing MetroCluster). The method of freezing the I/O is application dependent
Split the BCV/R1 devices in the secondary Symmetrix from the mirror group
# symmir -g <prisymdevgrpname> split -rdf
Resume the application I/O to the R1 devices in the primary Symmetrix if needed. The method of resuming the I/O is application dependent.
Incrementally establish the SRDF volume pairs between secondary Symmetrix and the recovery Symmetrix. The data is copied from BCV/R1 devices on the secondary Symmetrix to the R2 devices on the recovery Symmetrix.
# symrdf -g <prisymdevgrpname> est -rbcv
Use the following command to check the data synchronization from the BCV/R1 devices to the R2 devices. If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, the copying completed.
# symrdf -g <prisymdevgrpname> query -rbcv
Once the copy completes, split the SRDF link between the secondary Symmetrix and the recovery Symmetrix
# symrdf -g <prisymdevgrpname> split -rbcv
Incrementally establish the BCV devices in the recovery Symmetrix as mirrors of the standard devices if they are not already established. These BCV devices were fully established when the volume group were created.
# symmir -f <recbcvdev_textfile> -sid <recsymid> est
The <recbcvdev_textfile> is a file that contains a list of the standard device and BCV device pair in the recovery Symmetrix.
Re-establish the BCV/R1 devices in the secondary Symmetrix as mirrors of the standard devices:
# symmir -g <prisymdevgrpname> est -rdf

Refer to the sample script prirefreshrec in the /opt/cmcluster/toolkit/SGSRDF/cascade/Samples directory for examples of how these commands are used. This script is designed to be run only on a node that connects to the primary Symmetrix.

Data Refresh Procedures in the Steady State

Once the application starts writing data to the primary Symmetrix devices, the data on the recovery Symmetrix is out of sync with the primary data; the data is not current but consistent. The primary data needs to periodically synchronized to the recovery Symmetrix so its data is not too out of date. As long as the application continues writing new data to the primary Symmetrix, the data on the recovery Symmetrix will always be behind. The level of data currency on the recovery Symmetrix is dictated by the frequency at which it is refreshed. The refresh process is shown in Figure 8-9 “Data Refresh in Steady State”.

Figure 8-9 Data Refresh in Steady State

The following procedure describes the steps necessary to periodically copy the data from the secondary Symmetrix to the recovery Symmetrix while the application is running on the primary site.

Freeze the application I/O to prevent the application from writing data to the R1 devices in the primary Symmetrix. The method of freezing the I/O is application dependent.
Use the following command to check the state of the SRDF link from the R1 devices in the primary Symmetrix to the R2 devices in the secondary Symmetrix. Make sure that the state is "Synchronized" before executing step 3.
# symrdf -g <prisymdevgrpname> query
Split the BCV/R1 devices in the secondary Symmetrix from the mirror group:
# symmir -g <prisymdevgrpname> split -rdf
Resume the application I/O to the R1 devices in the primary Symmetrix if needed. The method of resuming the I/O is application dependent.
Split the BCV devices in the recovery Symmetrix from the mirror group. This is required to preserve an old copy of the data just in case a failure occurs during data synchronization between the secondary Symmetrix and the recovery Symmetrix that may cause data corruption on the R2 devices in the recovery Symmetrix.
# symmir -f <recbcvdev_textfile> -sid <recsymid> split
The <recbcvdev_textfile> is a file that contains a list of the standard device and BCV device pair in the recovery Symmetrix.
Incrementally establish the SRDF volume pairs between the secondary Symmetrix and the recovery Symmetrix. The data is copied from secondary Symmetrix BCV/R1 devices to the recovery Symmetrix R2 devices
# symrdf -g <prisymdevgrpname> est -rbcv
Use the following command to check the data synchronization from the BCV/R1 devices to the R2 devices. If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, the copying completed:
# symrdf -g <prisymdevgrpname> query -rbcv
Once the copy completes, split the SRDF link between the secondary Symmetrix and the recovery Symmetrix:
# symrdf -g <prisymdevgrpname> split -rbcv
Re-establish the BCV devices in the recovery Symmetrix as mirrors of the standard devices:
# symmir -f <recbcvdev_textfile> -sid <recsymid> est
Re-establish the BCV/R1 devices in the secondary Symmetrix as mirrors of the standard devices.
# symmir -g <prisymdevgrpname> est -rdf

Refer to the sample script prirefreshrec in the /opt/cmcluster/toolkit/SGSRDF/cascade/Samples directory for examples of how these commands are used. This script is designed to run only on a node that connects to the primary Symmetrix.

Data Replication in Failover and Failback Scenarios

This section describes the data replication procedures for various failover and failback scenarios.

Scenario 1—Primary Site within the Primary Cluster Fails

When a failure occurs at the primary site, the hosts are down or the whole site is down, the application package is automatically failover to the secondary site within the primary cluster. Until the problems at the primary site are fixed, and data replication is reestablished, there is no data protection for the package at the secondary site. Depending on the type of failure and how quickly the primary site is back online, data refresh to the recovery site is still needed. This scenario is illustrated in Figure 8-10 “Failure of Primary Site in Primary Cluster”.

Figure 8-10 Failure of Primary Site in Primary Cluster

Establishing Data Replication from the Secondary to the Recovery Site After Failover to the Secondary Site

After failover, the application is running on secondary site and writing I/O to R2 devices. The data is not remotely protected. The procedure to refresh the data from the secondary Symmetrix to the recovery Symmetrix is the same as the one that is done in steady state. But the procedure is now running on a system in the secondary site; therefore, the options on some of the SYMCLI commands are different.

Split the BCV devices in the recovery Symmetrix from the mirror group. This is required to preserve an old copy of the data just in case a failure occurs during data synchronization between the secondary Symmetrix and the recovery Symmetrix that may cause data corruption on the R2 devices in the recovery Symmetrix.
# symmir -f <recbcvdev_textfile> -sid <recsymid> split
The <recbcvdev_textfile> is a file that contains a list of the standard device and BCV device pair in the recovery Symmetrix.
Freeze the application I/O if the application is writing data to the R1 devices in the primary Symmetrix. The method of freezing the I/O is application dependent.
Split the BCV/R1 devices in the secondary Symmetrix from the mirror group and start synchronization with the R2 devices in the recovery Symmetrix:
# symmir -g <secsymdevgrpname> split
Resume the application I/O to the R1 devices in the primary Symmetrix if needed. The method of resuming the I/O is application dependent.
Incrementally establish the SRDF volume pairs between the secondary Symmetrix and the recovery Symmetrix. The data is synchronized from the secondary Symmetrix BCV/R1 devices to the recovery Symmetrix R2 devices
# symrdf -g <secsymdevgrpname> est -bcv
Use the following command to check the data synchronization from the BCV/R1 devices to the R2 devices. If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, the copying completed.
# symrdf -g <secsymdevgrpname> query -bcv
Once the copy completes, split the SRDF link between the secondary Symmetrix and the recovery Symmetrix:
# symrdf -g <secsymdevgrpname> split -bcv
Re-establish the BCV devices in the recovery Symmetrix as mirrors of the standard devices:
# symmir -f <recbcvdev_textfile> -sid <recsymid> est
Re-establish the BCV/R1 devices in the secondary Symmetrix as mirrors of the standard devices:
# symmir -g < secsymdevgrpname > est

Refer to the sample script secrefreshrec in the /opt/cmcluster/toolkit/SGSRDF/cascade/Samples directory for example of how these commands are used. This script is designed to run only on a node that connects to the secondary Symmetrix.

Failback from the Secondary Site to the Primary Site

Once the problems at the primary site have been fixed, the application can fail back to the primary site. The current RDF pair states of the package device groups will be "Split," which is not handled automatically by the package control script. The following steps are required to move the package back to the primary site.

Set the RDF pairs between primary Symmetrix and secondary Symmetrix to fail over state. Either use the following command on a system that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> failover -force
or use the following command on a system that connects to the primary Symmetrix
# symrdf -g <prisymdevgrpname> failover -force
This command can be run while the package application is still running. The device group has already been failed over, so all this command will do is change the RDF pair state from "Split" to "Failed Over."
Now, halt the application package:
# cmhaltpkg <package_name>
Split the BCV/R1 in the secondary Symmetrix from the mirror group to save a good copy of the data. From a system that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> split
From a system that connects to the primary Symmetrix:
# symmir -g <prisymdevgrpname> split -rdf
Start the application package on the primary site. Use the following command for all hosts in the primary cluster that may run this package:
# cmmodpkg -e -n <host_name> <package_name>
Then run the package with the following command:
# cmmodpkg -e <package_name>
The package will now start up on its primary host. The package control script will change the R1 devices from "Not Ready" status to "WD" (Write Disabled) status, then an RDF failback will be done on the package device group. The failback will synchronize the R1 devices from the R2 devices. Until the synchronization is complete, the package application may run at a lower performance level.
Use the following command to check the RDF pair state between the R1 devices in the primary Symmetrix to the R2 devices in the secondary Symmetrix. From a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> query
From a host that connects to the primary Symmetrix:
# symrdf -g <prisymdevgrpname> query
If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, then proceed to the next step
Re-establish the BCV/R1 devices in the secondary Symmetrix as a mirror of the standard device. From a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> est
From a host that connects to the primary Symmetrix:
# symmir -g <prisymdevgrpname> est -rdf

Scenario 2—Secondary Site within the Primary Cluster Fails

When the secondary site fails, or all SRDF links between the primary Symmetrix and the secondary Symmetrix fail, unless domino mode is used, the application running on the primary site is not aware of this failure and continues to run on the primary site. This scenario is illustrated in Figure 8-11 “Failure of Secondary Site in Primary Cluster”.

Figure 8-11 Failure of Secondary Site in Primary Cluster

Without the secondary site, the current configuration doesn't provide any means to replicate the new data from the primary Symmetrix directly to the recovery Symmetrix. If the secondary site is down for a long time, the data in the recovery Symmetrix is very out-of-date. If the primary site fails during this time, and the recovery takes over, the customer will have to operate on an old copy of the data. Therefore, it's important to fix and have the secondary site up and running as soon as possible.

When the secondary site is fixed, the SRDF volume pair between the primary Symmetrix and the secondary Symmetrix will be in "Suspended" mode. If the BCV/R1 in the secondary Symmetrix contains a good copy of the data, to protect this data from corruption in case of rolling disaster, these devices must be split from the mirror group before re-establishing the SRDF volume pairs between the primary Symmetrix and the secondary Symmetrix. Use the following steps:

Split the BCV/R1 devices in the secondary Symmetrix from the mirror group. From a host that connects to the primary Symmetrix:
# symmir -g <prisymdevgrpname> split -rdf
From a host that connects to the secondary Symmetrix:
# symmir -g <secisymdevgrpname> split
Incrementally establish the SRDF volume pairs between the primary Symmetrix and the secondary Symmetrix. From a host that connects to the primary Symmetrix:
# symrdf -g <prisymdevgrpname> est From a host that connects to the secondary Symmetrix:# symrdf -g <secsymdevgrpname> est
Use the following command to check the state of the SRDF link from the R1 devices in the primary Symmetrix to the R2 devices in the secondary Symmetrix. Make sure that the state is "Synchronized" before proceeding to the next step. From a host that connects to the primary Symmetrix:
# symrdf -g <prisymdevgrpname> query
From a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> query
Re-establish the BCV/R1 devices in the secondary Symmetrix as as a mirror of the standard device. From a host that connects to the primary Symmetrix:
# symmir -g <prisymdevgrpname> est -rdf
From a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> est

Scenario 3—Entire Primary Cluster Fails

In this scenario, the assumption is that both primary site and secondary site fail at the same time or very close to each other. This scenario is illustrated in Figure 8-12 “Failure of Entire Primary Cluster”.

Figure 8-12 Failure of Entire Primary Cluster

Failover from Primary Cluster to Recovery Cluster

After reception of the ContinentalClusters alerts and alarm, the administrators at the recovery site follow the prescribed processes and recovery procedures to start the protected applications on the recovery cluster. Note that data corruption may occur in situation where a disaster occurs at the primary cluster while the data refresh from secondary Symmetrix to the recovery Symmetrix is in progress. The data in the R2 devices in the recovery Symmetrix is not usable. The data can be recovered by restoring an old copy of the data from the BCV devices in the recovery Symmetrix. Execute the following commands to restore the data:

Restore the data from the BCV to the R2 devices in the recovery Symmetrix:
# symmir -g <recsymdevgrpname> -full restore
Use the following command to check the data restore progress. If the pair state is "Synchronized," the data restore is complete.
# symmir -g <recsymdevgrpname> query
Once the restore completed, split the BCV devices from the mirror group:
# symmir -g <recsymdevgrpname> split

The data in the recovery Symmetrix may not be current but should be consistent. There is no additional procedure needed. The package control script is programmed to handle this case.

After the application is up and running, re-establish the BCV devices as mirrors of the standard devices for an additional copy of the data:

# symmir -g <recsymdevgrpname> est

Failback from the Recovery Cluster to the Primary Site

The current configuration doesn't support the application failback to the primary site in the primary cluster unless the secondary site in the primary cluster is up and running. The secondary site has to be repaired first. The application can temporarily fail back to the secondary site while the primary site is still down. Before the application can fail back to either the secondary site or the primary site, the current data need to be restored from the recovery Symmetrix to the secondary Symmetrix and the primary Symmetrix.

Data Restore and Failback from the Recovery Cluster to the Secondary Site within the Primary Cluster

This procedure is used in situation where the application fails back and runs on the secondary site while the primary site is still down.

Halt the ContinentalClusters monitor package.
Halt the ContinentalClusters recovery packages at the recovery site.
Split the BCV/R1 from the mirror group if it's not already in the split state
The following example shows the command if run on a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> split
The <secbcvdev_textfile> is a file that contains a list of the standard device and BCV device pair in the secondary Symmetrix.
The following shows the command if run on a host that connects to the secondary Symmetrix:# symmir -g <secsymdevgrpname> split
Re-establish the RDF volume pairs between the recovery Symmetrix and the secondary Symmetrix. Restore the data from the recovery Symmetrix and the secondary Symmetrix. The following shows the command if run on a host that connects to the recovery Symmetrix:
# symrdf -g <recsymdevgrpname> -full restore
The following shows the command if run on a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> -full restore -bcv
Use the following command to check the data restore from the R2 devices in the recovery Symmetrix to the BCV/R1 devices in the secondary Symmetrix. If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, the data restore completed. The following shows the command if run on a host that connects to the recovery Symmetrix:
# symrdf -g <recsymdevgrpname> query The following shows the command if run on a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> query -bcv
Once the copy completes, split the SRDF link between secondary Symmetrix and recovery Symmetrix. The following shows the command if run on a host that connects to the recovery Symmetrix:
# symrdf -g <recsymdevgrpname> split
The following shows the command if run on a host that connects to the secondary Symmetrix:# symrdf -g <secsymdevgrpname> split -bcv
Re-establish the BCV devices in the secondary Symmetrix as mirrors of the standard devices. Restore the data from the BCV/R1 devices to the R2 standard devices. The following shows the command if run on a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> -full restore The following shows the command if run on a host that connects to the secondary Symmetrix:# symmir -g <secsymdevgrpname> -full restore
Use the following command to check the data restore from the BCV/R1 devices in the R2 standard devices in the secondary Symmetrix. If the "RDF Pair STATE" column shows the state "Restored" for all the devices, the data restore completed. The following shows the command if run on a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> query The following shows the command if run on a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> query
Once the data restore completed, split the BCV/R1 devices from the mirror group. The following shows the command if run on a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> split
The following shows the command if run on a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> split
Re-establish the BCV/R1 devices as mirrors of the standard devices. The following shows the command if run on a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> est
The following shows the command if run on a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> est
Since the recovery cluster has a different cluster ID, the data restore from the recovery Symmetrix to the secondary Symmetrix also copies the recovery cluster's ID to the secondary Symmetrix. Do the following to change the cluster's ID on each cluster aware volume group in the secondary Symmetrix before starting the application package:
# vgchange -c n /dev/<vg_name>
# vgchange -c y /dev/<vg_name>
Start the application package at the secondary site. Issue the following command for all hosts in secondary site that may run this package:
# cmmodpkg -e -n <host_name> <package_name>
Then issue the following command from any node on the primary cluster:
# cmmodpkg -e <package_name>
Start the ContinentalClusters monitor package.

Refer to the sample script recrestoresec in the /opt/cmcluster/toolkit/SGSRDF/cascade/Samples directory for the automation of step 3 to step 10. This script is designed to run only on a node that connects to the secondary Symmetrix.

Data Restore and Failback from the Recovery Site Directly to the Primary Site in the Primary Cluster

This procedure is used in situation where both the secondary site and the primary site are fixed and up and running. The package application fails back directly from the Recover cluster to the primary site in the primary cluster.

Halt the ContinentalClusters monitor package at the recovery site.
Halt the ContinentalClusters recovery packages at the recovery site.
Split the BCV/R1 from the mirror group if it's not already in the split state. Use the following command if run on a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> split
The <secbcvdev_textfile> is a file that contains a list of the standard device and BCV/R1 device pairs in the secondary Symmetrix.
Use the following command if run on a host that connects to the secondary Symmetrix:# symmir -g <secsymdevgrpname> split
Re-establish the RDF volume pairs between the recovery Symmetrix and the secondary Symmetrix. Restore the data from the recovery Symmetrix and the secondary Symmetrix. Use the following command if run on a host that connects to the recovery Symmetrix:
# symrdf -g <recsymdevgrpname> -full restore
Use the following command from a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> -full restore -bcv
Use the following command to check the data restore from the R2 devices in the recovery Symmetrix to the BCV/R1 devices in the secondary Symmetrix. If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, the data restore completed. From a host that connects to the recovery Symmetrix:
# symrdf -g <recsymdevgrpname> query
From a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> query -bcv
Once the restore completes, split the SRDF link between secondary Symmetrix and recovery Symmetrix. From a host that connects to the recovery Symmetrix:
# symrdf -g <recsymdevgrpname> split
From a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> split -bcv
Re-establish the BCV/R1 devices in the secondary Symmetrix as mirrors of the standard devices. Restore the data from the BCV/R1 devices to the R2 standard devices. From a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> -full restore
From a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> -full restore
Use the following command to check the data restore from the BCV/R1 devices in the R2 standard devices in the secondary Symmetrix. If the "RDF Pair STATE" column shows the state "Restored" for all the devices, the data restore completed. From a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> query
From a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> query
Once the data restore completed, split the BCV/R1 devices from the mirror group. From a host that connects to the recovery Symmetrix:
# symmir -f <secbcvdev_textfile> -sid <secsymid> split
From a host that connects to the secondary Symmetrix:# symmir -g <secsymdevgrpname> split
Set the RDF pairs between primary Symmetrix and secondary Symmetrix to fail over state. Run the following command on a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> failover -force
Since the recovery cluster has a different cluster ID, the data restore from the recovery Symmetrix to the secondary Symmetrix also copies the recovery cluster's ID to the secondary Symmetrix. Do the following to change the cluster's ID on each cluster aware volume group in the secondary Symmetrix before starting the application package:
# vgchange -c n /dev/<vg_name>
# vgchange -c y /dev/<vg_name>
Start the package application at the primary site. Issue the following command for all hosts in the primary cluster that may run this package:
# cmmodpkg -e -n <host_name> <package_name>
Start the package with the following command:
# cmmodpkg -e <package_name>
Start the ContinentalClusters monitor package on the recovery cluster.
Use the following command to check the RDF pair state between the R1 devices in the primary Symmetrix to the R2 devices in the secondary Symmetrix. From a host that connects to the secondary Symmetrix:
# symrdf -g <secsymdevgrpname> query
From a host that connects to the primary Symmetrix:# symrdf -g <prisymdevgrpname> query
If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, then proceed to the next step.
Re-establish the BCV/R1 devices in the secondary Symmetrix as mirrors of the standard devices. From a host that connects to the secondary Symmetrix:
# symmir -g <secsymdevgrpname> est
From a host that connects to the primary Symmetrix:
# symmir -g <prisymdevgrpname> est -rdf

Refer to the sample script recrestoresec in the /opt/cmcluster/toolkit/SGSRDF/cascade/Samples directory for the automation of steps 3 to step 10. This script is designed to be run only on a node that connects to the secondary Symmetrix.