Support for Oracle RAC instances means that the RAC instances running
on the primary cluster will be restarted by Continentalclusters on
the recovery cluster to continue serving the clients' databases requests
upon a primary cluster failure. Figure 4-11 “Oracle
RAC Instances in a Continentalclusters Environment” is a sample of Oracle RAC instances running in the Continentalclusters
environment.
As shown in the above example, Oracle RAC instances are configured
to run in Serviceguard packages. The instance packages are running
on the primary cluster and will be recovered on the recovery cluster
upon a primary cluster failure. Figure 4-12 “Sample
Oracle RAC Instances in a Continentalclusters Environment After Failover” shows a recovery using an Oracle RAC configuration after
failover.
Oracle RAC instances are only supported in the Continentalclusters environment
using SLVM for volume management, and using HP StorageWorks Continuous
Access XP (CA XP), or HP StorageWorks Continuous Access EVA (CA
EVA) or EMC Symmetrix Remote Data Facility (SRDF) for data replication.
To enable this feature, it is necessary to have either Metrocluster
with Continuous Access XP version A.04.20 (or later), or Metrocluster
with Continuous Access EVA version A.01.00 (or later), or Metrocluster
with SRDF version A.04.13 (or later) installed on all of the nodes
in the continental cluster.
Continentalclusters Oracle RAC support is available for a
cluster environment configured with only Serviceguard (for example,
the environment running with Oracle 9i), or a cluster environment configured
with Serviceguard plus Oracle Cluster Ready Service (for example,
the environment running with Oracle 10g).
Starting with Continentalclusters version A.05.01, recovery
of an Oracle RAC instance in a cluster environment running Serviceguard
and Oracle Cluster Ready Service (CRS) is supported. There is a
special configuration required for the environment running both
CRS and Serviceguard for the Continentalclusters RAC instance recovery protection.
For more information refer to the following section, “Required
Configuration for the Environment running with Serviceguard/Serviceguard
Extension for RAC and CRS (Oracle Cluster Software)”.
Configuring
the Environment for Continentalclusters to Support Oracle RAC |
 |
In order to enable Continentalclusters support for Oracle
RAC, there needs to be a set of configurations, which include either
CA XP, or CA EVA or EMC SRDF, Oracle RAC, and Continentalclusters.
To use this feature, Oracle RAC using either XP, or EVA or EMC disk
arrays and SLVM volume groups for storing the databases must have
already been configured. For replicating data between two clusters,
each of these must use either the CA XP, or CA EVA, or EMC SRDF.
Also, Continentalclusters must have already been installed
and configured in your environment. For complete installation and configuration
information of Oracle and HP Storage Works products, refer to the
HP StorageWorks and Oracle RAC manuals.
Use the following set of procedures to enable Continentalclusters recovery
support for Oracle RAC instances:
Configure either
CA XP, or CA EVA or EMC SRDF for data replication. SLVM volume groups
need to be configured on the disk arrays for storing Oracle database.
Volume group names configured on both primary and recovery clusters
must be the same. In addition, data replication needs to be setup
between the disk arrays associated with primary and recovery clusters.
For more details, refer to Chapter 5 “Building
Disaster-Tolerant Serviceguard Solutions Using Metrocluster with
Continuous Access XP” Chapter 6 “Building
Disaster Tolerant Serviceguard Solutions Using Metrocluster with
Continuous Access EVA” or Chapter 7 “Building
Disaster Tolerant Serviceguard Solutions Using Metrocluster with
EMC SRDF”
Configure Oracle RAC. All
the database files need to be configured to reside on the volume
groups configured in the previous step. The configuration of Oracle
RAC instances to be recovered in a Continentalclusters environment
need to be the same on both the primary and recovery clusters. For
more detail, refer to the Oracle RAC installation and
configuration manual. For the environment running with
CRS and Serviceguard, there are additional configuration requirements.
For more information refer to section, “Required
Configuration for the Environment running with Serviceguard/Serviceguard
Extension for RAC and CRS (Oracle Cluster Software)”.
Configure Continentalclusters.
Follow the steps described in section, “Building
the Continentalclusters Configuration”
Configure Oracle RAC instances
in Serviceguard Packages. Continentalclusters supports recovery
only for applications running in Serviceguard packages. In a multiple
recovery pair scenario, where more than one primary cluster share
the same recovery cluster, the primary RAC instance package name
must be unique on each primary cluster. For details on how to configure
an Oracle RAC instance in a Serviceguard package, refer to the
Using Serviceguard Extension for RAC user’s
guide.
In the Continentalclusters environment, one Oracle
RAC instance per Serviceguard package is required. Configure the
Oracle RAC instance packages on both primary and recovery clusters
based on the number of RAC instances configured to run on that cluster.
It is required that the same number of Oracle RAC instances are configured
on both primary and recovery clusters for the Continentalclusters
recovery protection.
The AUTO_RUN parameter in the package configuration file needs to be
set to “NO” similar to other applications packages
running in the Continentalclusters environment.
For the environment running with Serviceguard and CRS, use
the instance package configuration templates shipped with Continentalclusters
version A.05.01 or later under /opt/cmconcl/scripts for the instance package configuration. The templates
need to be modified before use in order to fit the target Oracle
environment.
Setup the environment file.
Instead of one environment file per continental cluster application
package, there is only one environment file for each set of Oracle
RAC instance packages accessing the same database. This file can
be located anywhere except the directory where the Oracle RAC instance
package configuration and control files reside. Only one environment
file can reside under one directory. The setup of the file is the
same as what is described in section, “Physical
Data Replication using Special Environment files” of this chapter, with the exception
of the PKGDIR variable.
The value of the PKGDIR variable should be the directory where this environment
file resides.
For specific information on how to setup the environment file
refer to Chapter 5 under section, “Configuring
Packages for Disaster Recovery” Chapter 6 under section, “Configuring
Packages for Automatic Disaster Recovery” or Chapter 7 under section “Configuring
Serviceguard Packages for Automatic Disaster Recovery”.
The environment file needs to be set up under the same path
on all the nodes of both the primary and recovery clusters in a
recovery pair. The following is the sample environment file, that
is (/etc/cmcluster/ccrac/db1/db1EnvFile_xpca.env) for the configuration using XP CA for data replication
between primary and recovery clusters:
AUTO_SVOLPFUS=0
AUTO_PSUSSSWS=0
AUTO_NONCURDATA=0
MULTIPLE_PVOL_OR_SVOL_FRAMES_FOR_PKG=0
export HORCMPERM=MGRNOINST
export HORCMINST=0
HORCTIMEOUT=360
WAITTIME=300
PKGDIR=”/etc/cmcluster/ccrac/db1”
FENCE=”async”
DEVICE_GROUP=”ccrac”
CLUSTER_TYPE=”continental” |
Setup of the Continentalclusters
Oracle RAC Specification File. The existence of file /etc/cmconcl/ccrac/ccrac.config serves as an enabler for Continentalclusters Oracle
RAC support. A template of this file is shipped with the product
in /opt/cmconcl/scripts. This file needs to be edited to fit your environment.
After editing, this file needs to be moved to /etc/cmconcl/ccrac/ccrac.config on all the nodes of all the participating clusters.
Use the following steps to set up the file:
Login
as root on one node of the primary cluster.
Change to your own directory:
# cd <your own directory>
Copy the file:
# cp /opt/cmconcl/scripts/ccrac.config \ ccrac.config.mycopy
Edit the file ccrac.config.mycopy to fit your environment.
The following parameters need to be edited:
CCRAC_ENV - fully qualified Metrocluster environment file name. This file naming convention as required by the Metrocluster software. It has to be appended with “_<DataReplication>.env” where <DataReplication> is the name of the data replication scheme being used. Refer to Metrocluster documents for the environment file naming convention.
This parameter is mandatory
CCRAC_SLVM_VGS - SLVM volume groups configured for the device specified in the above environment file for variable DEVICE_GROUP. These are the volume groups used by the associated RAC instance packages. It is important that all of the volume groups configured for the specified DEVICE_GROUP are listed. If only partial of the configured volume groups are listed, the device will not be prepared properly and the storage will result in an inconsistent state.
This parameter is mandatory.
CCRAC_INSTANCE_PKGS - the names of the configured RAC instance packages accessing in parallel the database stored in the specified volume groups.
This parameter is mandatory.
CCRAC_CLUSTER - Serviceguard cluster name configured as the primary cluster of the corresponding RAC instance package set.
This parameter is mandatory in the Continentalclusters configuration that contains more than two Serviceguard clusters, and is optional in the configuration that contains only two Serviceguard clusters.
CCRAC_ENV_LOG - logfile specification for the storage preparation output.
This parameter is optional. If not specified, ${CCRAC_ENV}.log will be used.
If multiple sets of Oracle instances accessing different databases are configured in your environment and need Continentalclusters recovery support, repeat the above set of parameters with an incremented index. For example,
CCRAC_ENV[0]=/etc/cmcluster/ccrac/db1 \
/db1EnvFile_xpca.env
CCRAC_SLVM_VGS [0]=ccracvg1 ccracvg2
CCRAC_INSTANCE_PKGS[0]=ccracpkg1 ccracpkg2
CCRAC_CLUSTER[0]= Cluster1
CCRAC_ENV_LOG[0]=/tmp/db1_prep.log |
CCRAC_ENV[1]=/etc/cmconcl/ccrac/db2 \
/db2EnvFile_xpca.env
CCRAC_SLVM_VGS[1]=ccracvg3 ccracvg4
CCRAC_INSTANCE_PKGS[1]=ccracPkg3 ccracPkg4
CCRAC_CLUSTER[1]= Cluster2
CCRAC_ENV_LOG[1]=/tmp/db2_prep.log |
CCRAC_ENV[2]=/etc/cmconcl/ccrac/db3 \
/db3EnvFile_xpca.env
CCRAC_SLVM_VGS[2]=ccracvg5 ccracvg6
CCRAC_INSTANCE_PKGS[2]=ccracPkg5 ccracPkg6
CCRAC_CLUSTER[2]= Cluster2 |
Copy the edited file to the
final directory:
# cp ccrac.config.mycopy \ /etc/cmconcl/ccrac/ccrac.config
Copy file /etc/cmconcl/ccrac/ccrac.config to all the other nodes of the cluster.
Login as root on one node of the recovery cluster and repeat steps “b” through “f” from
above. If the recovery cluster is configured to recover the Oracle
RAC instances for more than one primary cluster, the ccrac.config file on the recovery cluster should contain information
for all the primary clusters.
Configure Continentalclusters
Recovery Group for Oracle RAC instance. Define one recovery group
for each Oracle RAC instance recovery. The PRIMARY_PACKAGE specified for the Oracle RAC instance recovery group
is the name of the instance package configured on the primary cluster.
The RECOVERY_PACKAGE specified for the RAC instance recovery group is the
corresponding instance package name configured on the recovery cluster.
One recovery group for each RAC instance is required in the Continentalclusters configuration
file. For example:
|
RECOVERY_GROUP_NAME instanceRG1
PRIMARY_PACKAGE ClusterA/instancepkg1
RECOVERY_PACKAGE ClusterB/instancepkg1'
RECOVERY_GROUP_NAME instanceRG2
PRIMARY_PACKAGE ClusterA/instancepkg2
RECOVERY_PACKAGE ClusterB/instancepkg2' |
Packages instancepkg1 and instancepkg2 are configured to run on primary cluster “ClusterA”. Packages instancepkg1 and instancepkg2 are configured to be restarted or recovered on the recovery
cluster “ClusterB” upon primary cluster failure.
After editing the Continentalclusters configuration file to
add in the recovery group specification for Oracle RAC instance
packages, it is required to manually apply the new configuration
by running the cmapplyconcl command.
When configuration in a recovery pair with RAC support is
finished, your systems should have sets of files similar to those
shown in Figure 4-13 “Continentalclusters Configuration
Files in a Recovery Pair with RAC Support”.
Required
Configuration for the Environment running with Serviceguard/Serviceguard
Extension for RAC and CRS (Oracle Cluster Software) |
|
The following are the required configurations for Continentalclusters RAC
instance recovery support for the cluster environment running with Serviceguard/Serviceguard
Extension for RAC and CRS (Oracle Cluster Software):
The Oracle
RAC environment running with Serviceguard/Serviceguard Extension
for RAC and Oracle Cluster Software should follow all the recommendations
listed in the Serviceguard and SGeRAC manuals for running with CRS
(Oracle Cluster Software).
CRS should not activate the
volume groups configured for the database automatically at startup
time. The file /var/opt/oracle/oravg.conf should not exist on any node of the primary and recovery
cluster.
The CRS storage (OCR and
voting disk) should be configured on a separate volume group than
the ones for the databases which are to be accessed by the RAC instances.
The RAC instances configured
to be recovered by Continentalclusters should be disabled with CRS
after they are created on the recovery cluster.
# srvctl disable instance -i instanceName
The RAC instance attribute
AUTO_START listed in the CRS service profile should be set to 2
on both primary and recovery clusters so that the instance will
not be automatically started when the node rejoins the cluster.
Login as the oracle administrator and use the following
steps to change the attribute value:
Generate
the resource profile.
crs_stat -p instance_name > $CRS_HOME/crs/public/instance_name.cap
Edit the resource profile
and set AUTO_START value to 2.
Register the value.
crs_register -u instance_name
Verify the value.
crs_stat -p instance_name
Initial
Startup of Oracle RAC Instance in a Continentalclusters Environment |
|
To ensure that the disk array will be ready for access in
shared mode for the Oracle RAC instances, it is recommended that
the user runs the Continentalclusters tool /opt/cmconcl/bin/ccrac_mgmt.ksh to initially startup the configured instance packages.
This tool ensures that the configured disk array will be ready in
writable mode for shared access before starting up the RAC instance
packages. If this tool is not used, manual checking is needed to
make sure the storage is ready in writable and shared access mode
before starting the RAC instance packages.
|
| |
|
 | NOTE: It is recommended that ccrac_mgmt.ksh is used for the initial startup of the RAC instance
package, or for failing back the RAC instance packages. This tool
should not be used at the recovery site for recovering RAC instance
packages, cmrecovercl is used in this case. |
|
| |
|
After the initial startup, use Serviceguard commands cmhaltpkg, cmrunpkg, cmmodpkg as needed to halt and restart the packages on the primary
cluster.
Use the following steps on any node of the primary cluster
to do the initial startup of the Oracle RAC instance packages:
If the cluster
is running with Serviceguard and Oracle CRS, make sure that the
CRS daemons and the required Oracle services, such as listener,
GSD, ONS, and VIP are up and running on all the nodes the RAC database
instances are configured to run.
Make sure /etc/cmconcl/ccrac/ccrac.config exists and was edited to contain the appropriate information.
To start all the RAC instance
packages configured to run as primary packages on the local cluster.
# /opt/cmconcl/bin/ccrac_mgmt.ksh start
To start a specific set of RAC instance
packages.
# /opt/cmconcl/bin/ccrac_mgmt.ksh -i <indexNumber> start
<IndexNumber> is the index used in the /etc/cmconcl/ccrac/ccrac.config file for the target set of the Oracle RAC instance packages.
To stop all the RAC instance
packages configured to run as primary packages on the local cluster.
# /opt/cmconcl/bin/ccrac_mgmt.ksh stop
To stop a specific set of RAC instance
packages.
# /opt/cmconcl/ccrac_mgmt.ksh -i <indexNumber> stop
<IndexNumber> is the index used in the /etc/cmconcl/ccrac/ccrac.config file for the target set of the Oracle RAC instance packages.
Failover
of Oracle RAC Instances to the Recovery Site |
|
Upon a disaster that disables the primary cluster, to start
up a Continentalclusters recovery process.
# cmrecovercl
For the cluster environment running with Serviceguard and
CRS, confirm that the CRS daemons and the required Oracle services,
such as listener, GSD, ONS, and VIP, are started on all the nodes,
which the database instance are configured to run before initiating
the recovery process.
|
| |
|
| NOTE: Make sure that the primary site is unavailable and all
of the Oracle RAC instance packages are not running on the primary
cluster before initiating the recovery process. |
|
| |
|
The Continentalclusters command, cmrecovercl prepares the configured storage for Oracle RAC instances
shared access only when the file /etc/cmconcl/ccrac/ccrac.config exists. If this file does not exist, the configured
storage will not be prepared for shared access before recovering
the Oracle RAC instance packages. If Continentalclusters recovery
group configuration includes Oracle RAC instance packages, these
packages will not be able to start or operate successfully.
The recovery process will startup the configured Oracle RAC
instance packages as well as other application packages configured
in the Continentalclusters environment.
If the Continentalclusters Oracle RAC support is enabled (the /etc/cmconcl/ccrac/ccrac.config file exists), the following messages will be prompted
to the user when the command cmrecovercl is invoked and confirmations are needed for the process
to proceed.
WARNING: this command will take over for the primary cluster Primary_Cluster_Name by starting the recovery package on the recovery cluster Recovery_Cluster_Name. You must follow your site disaster recovery procedure to ensure that the primary packages on Primary_Cluster_Name are not running and that recovery on Recovery_Cluster_Name is necessary. Continue with this command while the applications are running on the primary cluster may result in data corruption. Be sure that the primary packages are not running and will not come back, and to start the recovery packages [y/n]?
The configuration file /etc/cmconcl/ccrac/ccrac.config for Oracle RAC instance support exists. Data storage specified in the file for this cluster will be prepared for this recovery process. If you choose “n” - not to prepare the storage for this recovery process, make sure that the required storage for this recovery process has been properly prepared.
The Oracle RAC instance package can be started in sequence.
# cmrecovercl -g <recoverygroupname>
Option -g is used to start up the first instance package, wait
until the disk arrays are synchronized before starting up the second
instance package.
If option -g is used with the command cmrecovercl, the following messages will be given instead:
WARNING: This command will take over for the primary cluster Primary_Cluster_Name by starting the recovery package on the recovery cluster Recovery_Cluster_Name. You must follow your site disaster recovery procedure to ensure that the primary packages on Primary_Cluster_Name are not running and that recovery on Recovery_Cluster_Name is necessary. Continuing with this command while the applications are running on the primary cluster may result in data corruption.
Are you sure that the primary packages are not running and will not come back, and are you certain that you want to start the recovery packages [y/n]?
The configuration file /etc/cmconcl/ccrac/ccrac.config for Oracle RAC instance support exists. If the primary package in the target group is configured within this file, the corresponding data storage will be prepared before starting the recovery package. If you choose (“n”) - not to prepare the storage for this recovery process, make sure that the required storage for this recovery process has been properly prepared.
Is this what you intend to do [y/n].
These message prompts can be disabled by running cmrecovercl with option -y.
|
| |
|
| NOTE: At the recovery time, Continentalclusters is responsible
for recovering the Oracle RAC instance packages configured. The
data integrity and currency at the recovery site are based on your
data replication configuration in the Oracle environment. |
|
| |
|
Failback
of Oracle RAC Instances After a Failover |
|
After failover, the configured disk array at the old recovery
cluster becomes the primary storage of the database. The Oracle
RAC instances are running at the recovery cluster after a successful
recovery. To failback the Oracle RAC instances to the primary cluster,
follow the procedures listed below:
Before failing back the Oracle RAC instances, make sure that
the data in the original primary site disk array is in an appropriate
state. Follow the disk array specific procedures for data resynchronization
between two clusters, and the Oracle RAC failback procedures before
restarting the instance.
|
| |
|
| NOTE: Make sure the AUTO_RUN flag for all the configured Continentalclusters packages
is disabled before restarting the cluster. |
|
| |
|
Fix the problems
that caused the primary site failure.
Stop the Oracle RAC instance
packages running on the recovery cluster. On any node of the recovery
cluster.
# /opt/cmconcl/bin/ccrac_mgmt.ksh stop
Synchronize the data between
the two participating clusters. Make sure that the data integrity
and the data currency are at the expected level at the primary site.
Verify that the primary cluster
is up and running. On any node of the primary cluster.
# cmviewcl
If the cluster is running
with Serviceguard and Oracle CRS, make sure that CRS and the required
services, such as listener, GSD, ONS, and, VIP are up and running
on all of the instance nodes. By default, when CRS is started, these
Oracle services are initiated.
Startup the Oracle RAC instance
packages on the primary cluster. On any node of the primary cluster.
# /opt/cmconcl/bin/ccrac_mgmt.ksh start
This command fails back all of the RAC instance packages configured to
adopt to this cluster as the primary cluster.
To failback only a specific set of the
Oracle RAC instance package set.
# /opt/cmconcl/bin/ccrac_mgmt.ksh [-i <indexNumber>] \ start
<indexNumber> is the index used in the /etc/cmconcl/ccrac/ccrac.config file for the target set of the Oracle RAC instance packages.
Printable version
