Support for Oracle RAC instances means the RAC instances running
on the primary cluster will be restarted by Continentalclusters
on the recovery cluster to continue serving the clients' databases
requests upon a primary cluster failure. Figure 4-11 “Oracle
RAC Instances in a Continentalclusters Environment” is a sample of Oracle RAC instances running in
the Continentalclusters environment.
As shown in the above example, Oracle RAC instances are configured
to run in Serviceguard packages. The instance packages are running
on the primary cluster and will be recovered on the recovery cluster
upon a primary cluster failure. Figure 4-12 “Sample
Oracle RAC Instances in a Continentalclusters Environment After Failover” shows a recovery using an Oracle RAC configuration after
failover.
The Oracle RAC instances are only supported in the Continentalclusters environment
using SLVM for volume management, and using HP StorageWorks Continuous
Access XP (CA XP) or EMC Symmetrix Remote Data Facility (SRDF) for
data replication. This feature is targeted for the customers who
use HP StorageWorks XP or EMC disk arrays to store the Oracle database
and use CA XP or EMC/SRDF to replicate the data changes between
two clusters. To enable this feature, you should have Metrocluster
with Continous Access XP version A.04.20 (or later) or Metrocluster
with SRDF version A.04.13 (or later) installed on all of the nodes
in the continental cluster.
Configuring
the Environment for Continentalclusters to Support Oracle RAC |
 |
In order to enable Continentalclusters support for Oracle
RAC, there needs to be a set of configurations, which include CA
XP or EMC SRDF, Oracle RAC, and Continentalclusters. To use this
feature, it is assumed that you have already configured Oracle RAC
using XP or EMC disk arrays and SLVM volume groups for storing the
databases and use the CA XP or EMC/SRDF for replicating data between
two clusters. Also, it is assumed that you have already installed
and configured Continentalclusters in your environment. Refer to
the HP StorageWorks and Oracle RAC manuals for the complete installation
and configuration information of these products.
Use the following set of procedures to enable Continentalclusters recovery
support for Oracle RAC instances:
Configure XP/CA
or EMC/SRDF for data replication. SLVM volume groups need to be
configured on the disk arrays for storing Oracle database. Volume
group names configured on both primary and recovery clusters must
be the same. In addition, data replication needs to be setup between
the disk arrays associated with primary and recovery clusters.
For more details, refer to Chapter 5 “Building
Disaster-Tolerant Serviceguard Solutions Using Metrocluster with
Continuous Access XP” or Chapter 7 “Building
Disaster Tolerant Serviceguard Solutions Using Metrocluster with
EMC SRDF”
Configure Oracle RAC. All
the database files need to be configured to reside on the volume
groups configured in the previous step. The configuration of Oracle
RAC instances to be recovered in a Contintenalclusters environment
need to be the same on both the primary and recovery clusters. For
more detail, refer to the Oracle RAC installation and
configuration manual.
Configure Continentalclusters.
Follow the steps described in this chapter to configure Continentalclusters.
Configure Oracle RAC instances
in Serviceguard Packages. Continentalclusters supports recovery
only for applications running in Serviceguard packages. In a multiple
recovery pair scenario, where more than one primary cluster share
the same recovery cluster, the primary RAC instance package name
must be unique on each primary cluster. For details on how to configure
an Oracle RAC instance in a Serviceguard package, refer to the
Using Serviceguard Extension for RAC manual.
In the Continentalclusters environment, one Oracle
RAC instance per Serviceguard package is required. Configure the
Oracle RAC instance packages on both primary and recovery clusters
based on the number of RAC instances configured to run on that cluster.
It is required that the same number of Oracle RAC instances are configured
on both primary and recovery clusters.
The AUTO_RUN parameter in the package configuration file needs to
be set to “NO” similar to other applications packages
running in the Continentalclusters environment.
Setup environment file. Instead
of one environment file per Continentalcluster application package
there is only one environment file for each set of Oracle RAC instance
packages accessing the same database. This file can be located
anywhere except the directory where the Oracle RAC instance package configuration
and control files reside. Only one environment file can reside under
one directory. The setup of the file is the same as what is described
in section, “Physical
Data Replication using Special Environment files” of
this chapter, except the PKGDIR variable.
The value of PKGDIR variable should be the directory
where this environment file resides.
For specific information on how to setup the environment file
refer to Chapter 5 under section, “Configuring
Packages for Disaster Recovery” or Chapter 7 under section “Configuring
Serviceguard Packages for Automatic Disaster Recovery”.
The environment file needs to be set up under the same path
on all the nodes of both the primary and recovery clusters in a
recovery pair. The following is the sample environment file for
(e.g. /etc/cmcluster/ccrac/db1/db1EnvFile_xpca.env) the configuration using XP CA for data replication
between primary and recovery clusters:
AUTO_SVOLPFUS=0
AUTO_PSUSSSWS=0
AUTO_NONCURDATA=0
MULTIPLE_PVOL_OR_SVOL_FRAMES_FOR_PKG=0
export HORCMPERM=MGRNOINST
export HORCMINST=0
HORCTIMEOUT=360
WAITTIME=300
PKGDIR=”/etc/cmcluster/ccrac/db1”
FENCE=”async”
DEVICE_GROUP=”ccrac”
CLUSTER_TYPE=”continental” |
Setup Continentalclusters
Oracle RAC Specification File. The existence of file /etc/cmconcl/ccrac/ccrac.config serves as an enabler for Continentalclusters Oracle
RAC support. A template of this file is shipped with the product
in /opt/cmconcl/scripts. This file needs to be edited to fit your environment.
After editing, this file needs to be moved to /etc/cmconcl/ccrac/ccrac.config on all the nodes of all the participating clusters.
To set up the file use the following steps:
Login
as root on one node of the primary cluster.
Change to your own directory:
# cd <your own directory>
Copy the file:
# cp /opt/cmconcl/scripts/ccrac.config \
ccrac.config.mycopy
Edit ccrac.config.mycopy
to fit your environment.
The following parameters need to be edited:
CCRAC_ENV -fully qualified Metrocluster environment file name. This file naming convention as required by the Metrocluster software. It has to be appended with “_<DataReplication>.env” where <DataReplication> is the name of the data replication scheme being used. Refer to Metrocluster chapters for the environment file naming convention. This parameter is mandatory. |
|
|
CCRAC_SLVM_VGS - SLVM volume groups configured for the device specified in the above environment file for variable DEVICE_GROUP. It is important that all the volume groups configured for the specified DEVICE_GROUP are listed.CCRAC_INSTANCE_PKGS - the names of the configured Oracle RAC instance packages, on the primary cluster, accessing in parallel the database stored in the specified volume groups.CCRAC_CLUSTER - Serviceguard cluster name configured as the primary cluster of the corresponding RAC instance package set. This parameter is mandatory in the Continentalclusters configuration that contains more than two Serviceguard clusters, and is optional in a configuration that contains two Serviceguard clusters. |
|
|
CCRAC_ENV_LOG - the logfile specification for the storage preparation output. This parameter is optional. If not specified, ${CCRAC_ENV}.log will be used.If multiple sets of Oracle 9i instances accessing different databases are configured in your environment and need Continentalclusters recovery support, repeat the above set of parameters with an incremented index. For example, |
CCRAC_ENV[0]=/etc/cmcluster/ccrac/db1 \/db1EnvFile_xpca.envCCRAC_SLVM_VGS [0]=ccracvg1 ccracvg2CCRAC_INSTANCE_PKGS[0]=ccracpkg1 ccracpkg2 |
CCRAC_CLUSTER[0]= Cluster1 |
CCRAC_ENV_LOG[0]=/tmp/db1_prep.log#CCRAC_ENV[1]=/etc/cmconcl/ccrac/db2 \/db2EnvFile_xpca.envCCRAC_SLVM_VGS[1]=ccracvg3 ccracvg4CCRAC_INSTANCE_PKGS[1]=ccracPkg3 ccracPkg4 |
CCRAC_CLUSTER[1]= Cluster2 |
CCRAC_ENV_LOG[1]=/tmp/db2_prep.log#CCRAC_ENV[2]=/etc/cmconcl/ccrac/db3 \/db3EnvFile_xpca.envCCRAC_SLVM_VGS[2]=ccracvg5 ccracvg6 |
CCRAC_INSTANCE_PKGS[2]=ccracPkg5 ccracPkg6 |
CCRAC_CLUSTER[2]= Cluster2 |
Copy the edited file to the final directory:
# cp ccrac.config.mycopy \ /etc/cmconcl/ccrac/ccrac.config
Copy file /etc/cmconcl/ccrac/ccrac.config to all the other nodes of the cluster.
Login as root on one node
of the recovery cluster and repeat steps b through f from above.
If the recovery cluster is configured to recover the Oracle RAC
instances for more than one primary cluster, the ccrac.config file
on the recovery cluster should contain information for all the primary
clusters.
Configure Continentalclusters
Recovery Group for Oracle RAC instance. Define one recovery group
for each Oracle RAC instance recovery. The PRIMARY_PACKAGE specified
for the Oracle RAC instance recovery group is the name of the instance
package configured on the primary cluster. The RECOVERY_PACKAGE specified
for the RAC instance recovery group is the corresponding instance
package name configured on the recovery cluster. One recovery group
for each RAC instance is required in the Continentalclusters configuration
file. For example:
|
RECOVERY_GROUP_NAME instanceRG1 PRIMARY_PACKAGE ClusterA/instancepkg1 RECOVERY_PACKAGE ClusterB/instancepkg1'RECOVERY_GROUP_NAME instanceRG2 PRIMARY_PACKAGE ClusterA/instancepkg2 RECOVERY_PACKAGE ClusterB/instancepkg2' |
Packages instancepkg1 and instancepkg2 are configured to run on primary cluster “ClusterA”.
Packages instancepkg1' and instancepkg2' are configured to be restarted or recovered on recovery cluster “ClusterB” upon
primary cluster failure.
After editing the Continentalclusters configuration file to
add in the recovery group specification for Oracle RAC instance
packages, you have to manually apply the new configuration by running
the command, cmapplyconcl.
When configuration in a recovery pair with RAC support is
finished, your systems should have sets of files similar to those
shown in Figure 4-13 “Continentalclusters Configuration
Files in a Recovery Pair with RAC Support”.
Initial
Startup of Oracle RAC Instance in a Continentalclusters Environment |
|
To ensure that the disk array will be ready for the access
in shared mode for the Oracle RAC instances, it is recommended the
user runs the Continentalclusters tool /opt/cmconcl/bin/ccrac_mgmt.ksh to initially startup the configured instance packages.
This tool ensures that the configured disk array will be ready in
writeable mode for shared access before starting up the RAC instance
packages. If this tool is not used, manual checking to make sure
the storage is ready in writeable and shared access mode is needed
before starting the RAC instance packages.
|
| |
|
 | NOTE: It is recommended that ccrac_mgmt.ksh is used for
the initial startup of the RAC instance package, or for failing
back the RAC instance packages. This tool should not be used at
the recovery site for recovering RAC instance packages, cmrecovercl
is used in this case. |
|
| |
|
After the initial startup you may use Serviceguard commands cmhaltpkg,
cmrunpkg, cmmodpkg as needed to halt and restart the packages on
the primary cluster.
Use the following steps on any node of the primary cluster
to do the initial startup of the Oracle RAC instance packages:
Make sure
/etc/cmconcl/ccrac/ccrac.config exists and was edited to contain the appropriate information.
To start all the RAC instance
packages configured to run as primary packages on the local cluster,
use the command:
# /opt/cmconcl/bin/ccrac_mgmt.ksh start
To start a specific set of RAC instance
packages use the command:
# /opt/cmconcl/bin/ccrac_mgmt.ksh -i <indexNumber> start
<IndexNumber> is the index used in the /etc/cmconcl/ccrac/ccrac.config file for the target set of the Oracle RAC instance packages.
To stop all the RAC instance
packages configued to run as primary packages on the local cluster,
use the command:
# /opt/cmconcl/bin/ccrac_mgmt.ksh stop
To stop a specific set of RAC instance
packages use the command:
# /opt/cmconcl/ccrac_mgmt.ksh -i <indexNumber> stop
<IndexNumber> is the index used in the /etc/cmconcl/ccrac/ccrac.config file for the target set of the Oracle RAC instance packages.
Failover
of Oracle RAC Instances to the Recovery Site |
|
Upon a disaster that disables the primary cluster, use the
following command to start up a Continentalclusters recovery process:
# cmrecovercl
|
| |
|
| NOTE: Make sure that the primary site is unavailable and all
of the Oracle RAC instance packages are not running on the primary
cluster before triggering the recovery process. |
|
| |
|
The Continentalclusters command, cmrecovercl prepares the configured storage for Oracle RAC instances
shared access only when the file /etc/cmconcl/ccrac/ccrac.config exists. If this file does not exist, the configured
storage will not be prepared for shared access before recovering
the Oracle RAC instance packages. If Continentalclusters recovery
group configuration includes Oracle RAC instance packages, these
packages will not be able to start or operate successfully.
The recovery process will startup the configured Oracle RAC
instance packages as well as other application packages configured
in the Continentalclusters environment.
If the Continentalclusters Oracle RAC support is enabled (/etc/cmconcl/ccrac/ccrac.config
file exists), the following messages will be prompted to the user
when the command cmrecovercl is invoked and confirmations are needed for the process
to proceed:
“WARNING: This command will take over for the primary
cluster Primary_Cluster_Name by starting the recovery package on
the recovery cluster Recovery_Cluster_Name. You must follow your
site disaster recovery procedure to ensure that the primary packages
on Primary_Cluster_Name are not running and that recovery on Recovery_Cluster_Name
is necessary. Continuing with this command while the applications
are running on the primary cluster may result in data corruption.
Are you sure that the primary packages are not running and will
not come back, and are you certain that you want to start the recovery
packages [y/n]?
The configuration file /etc/cmconcl/ccrac/ccrac.config for Oracle RAC instances support exists. Data storage
specified in the file will be prepared for this recovery process.
If you choose not to prepare the storage for this recovery process,
make sure that the required storage for this recovery process has
been properly prepared. Is this what you intend to do [y/n]?
The Oracle RAC instance package can be started in sequence
by using:
# cmrecovercl -g <recoverygroupname>
Option -g is used to start up the first instance package,
you should wait until the disk arrays are synchronized before starting
up the second instance package.
If option “-g”is used with the command cmrecovercl, the following messages will be given instead:
“WARNING: This command will take over for the primary
cluster Primary_Cluster_Name by starting the recovery package on
the recovery cluster Recovery_Cluster_Name. You must follow your
site disaster recovery procedure to ensure that the primary packages
on Primary_Cluster_Name are not running and that recovery on Recovery_Cluster_Name
is necessary. Continuing with this command while the applications
are running on the primary cluster may result in data corruption.
Are you sure that the primary packages are not running and will
not come back, and are you certain that you want to start the recovery
packages [y/n]?
The configuration file /etc/cmconcl/ccrac/ccrac.config for Oracle RAC instances support exists. If the primary
package in the target group is configured within this file, the
corresponding data storage will be prepared before starting the
recovery pacakge. If you choose not to prepare the storage for the
recovery process, make sure that the required storage for the recovery
package has been properly prepared. Is this what you intend to do
[y/n]?
These message prompts can be disabled by running cmrecovercl with option “-y”.
|
| |
|
| NOTE: At the recovery time, Continentalclusters is responsible
for recovering the Oracle RAC instance packages configured. The
data integrity and data currency at the recovery site are based
on your data replication configuration in the Oracle environment. |
|
| |
|
Failback
of Oracle RAC Instances After a Failover |
|
After failover, the configured disk array at the old recovery
cluster becomes the primary storage of the database. The Oracle
RAC instances are running at the recovery cluster after a successful
recovery. To failback the Oracle RAC instances to the primary cluster,
follow the procedures listed below:
Before failing back the Oracle RAC instances, make sure the
data in the original primary site disk array is in an appropriate
state. Follow the disk array specific procedures for data resynchronization
between two clusters, and the Oracle RAC failback procedures before
restarting the instance.
|
| |
|
| NOTE: Make sure the AUTO_RUN flag for all the configured Continentalclusters
packages is disabled before restarting the cluster. |
|
| |
|
Fix the problems
that caused the primary site failure.
Stop the Oracle RAC instance
packages running on the recovery cluster. On any node of the recovery
cluster run the command: #/opt/cmconcl/bin/ccrac_mgmt.ksh stop
Synchronize the data between
the two participating clusters. Make sure that the data integrity
and data currency are at the expected level at the primary site.
Verify that the primary cluster
is up and running. On any node of the primary cluster run the command:
# cmviewcl
Startup the Oracle RAC instance
packages on primary cluster. On any node of the primary cluster
run the command: #/opt/cmconcl/bin/ccrac_mgmt.ksh start
This command fails back all of the RAC instance packages configured
to adopt to this cluster as the primary cluster.
If you want to failback only a specific set of
the Oracle RAC instance package set, do the following:
# /opt/cmconcl/bin/ccrac_mgmt.ksh [-i <indexNumber>] start
<indexNumber> is the index used in the /etc/cmconcl/ccrac/ccrac.config file for the target set of the Oracle RAC instance packages.
Printable version
