Using Packages to Configure Startup and Shutdown of RAC Instances

To automate the startup and shutdown of RAC instances on the nodes of the cluster, you can create packages which activate the appropriate volume groups and then run RAC. Refer to the section “Creating Packages to Launch Oracle RAC Instances”




	NOTE: The maximum number of RAC instances is 127 per cluster

Starting Oracle Instances

Once the Oracle installation is complete, ensure that all package control scripts are in place on each node and that each /etc/rc.config.d/cmcluster script contains the entry AUTOSTART_CMCLD=1. Then reboot each node. Within a couple of minutes following reboot, the cluster will reform, and the package control scripts will bring up the database instances and application programs.

When Oracle has been started, you can use the SAM process management area or the ps -ef command on both nodes to verify that all RAC daemons and Oracle processes are running.

Starting Up and Shutting Down Manually

To start up and shut down RAC instances without using packages, you can perform the following steps.

Starting up involves the following sequence:
1. Start up the cluster (cmrunnode or cmruncl)
2. Activate the database volume groups or disk groups in shared mode.
3. Bring up Oracle in shared mode.
4. Bring up the Oracle applications, if any.
Shutting down involves the following sequence:
1. Shut down the Oracle applications, if any.
2. Shut down Oracle.
3. Deactivate the database volume groups or disk groups.
4. Shut down the cluster (cmhaltnode or cmhaltcl).

If the shutdown sequence described above is not followed, cmhaltcl or cmhaltnode may fail with a message that GMS clients (RAC 9i) are active or that shared volume groups are active.

Creating Packages to Launch Oracle RAC Instances

To coordinate the startup and shutdown of RAC instances with cluster node startup and shutdown, you create a one-node package for each node that runs an RAC instance. In the package configuration file, you should specify only the single node on which the instance will run and specify the control script that is to be executed every time the instance node or the entire RAC cluster starts up or shuts down.




	NOTE: You must create the RAC instance package with a `PACKAGE_TYPE` of `FAILOVER`, but the fact that you are entering only one node ensures that the instance will only run on that node.

To simplify the creation of RAC instance packages, you can use the Oracle template provided with the separately purchasable ECM Toolkits product (B5139BA). This file is found in /opt/cmcluster/tookit/DB/Oracle. Use the special oracle.sh script that is provided, and follow the instructions that appear in the README file. Also refer to the section “Customizing the Control Script for RAC Instances,” below.

To create the package with Serviceguard Manager select the cluster. From the actions menu, choose configure package. To modify a package, select the package. For an instance package, create one package for each instance. On each node, supply the SID name for the package name.

To create a package on the command line, use the cmgetconf command to get an editible configuration file.

Set the AUTO_RUN parameter to YES, if you want the instance to start up as soon as the node joins the cluster. In addition, you should set the NODE_FAILFAST_ENABLED parameter to NO.

If you are using CVM disk groups for the RAC database, be sure to include the name of each disk group on a separate STORAGE_GROUP line in the configuration file.

Configuring Packages that Access the Oracle RAC Database

You can also use packages to start up applications that access the RAC instances. If an application is intended to fail over among cluster nodes, then you must set it up as a distinct package, separate from the package that starts and stops the RAC instance. Use the following procedures for packages that contain applications which access the RAC database:

In the ASCII package configuration file, set the AUTO_RUN parameter to NO, or if you are using Serviceguard Manager to configure packages, set Automatic Switching to Disabled. This keeps the package from starting up immediately when the node joins the cluster, and before RAC is running.
You can then manually start the package using the cmmodpkg -e packagename command after RAC is started. Alternatively, you can choose to automate the process of package activation by writing your own script, and copying it to all nodes that can run the package. This script should contain the cmmodpkg -e command and activate the package after RAC and the cluster manager have started.

Adding or Removing Packages on a Running Cluster

You can add or remove packages while the cluster is running, subject to the limit of MAX_CONFIGURED_PACKAGES. To add or remove packages online, refer to the section “Cluster and Package Maintenance” in chapter 7 of the Managing Serviceguard user’s guide.

Writing the Package Control Script

The package control script contains all the information necessary to run all the services in the package, monitor them during operation, react to a failure, and halt the package when necessary. You can use either Serviceguard Manager or HP-UX commands to create or modify the package control script. For security reasons, the control script must reside in a directory with the string cmcluster in the path.

Using Serviceguard Manager to Write the Package Control Script

As you complete the tabs for the configuration, the control script can be generated automatically. When asked to supply the the pathname of the package run and halt scripts, use the filenames from the ECM toolkit. For more information, use the Help key.

When you create a package control script this way, you do not need to do any further editing, but you may customize the script if you wish.

Using Commands to Write the Package Control Script

Each package must have a separate control script, which must be executable. The control script is placed in the package directory and is given the same name as specified in the RUN_SCRIPT and HALT_SCRIPT parameters in the package ASCII configuration file. The package control script template contains both the run instructions and the halt instructions for the package. You can use a single script for both run and halt operations, or, if you wish, you can create separate scripts.

Use the following procedure to create a control scripts for the sample package pkg1.

First, generate a control script template:

# cmmakepkg -s /etc/cmcluster/pkg1/control.sh

You may customize the script, as described in the section, ““Customizing the Package Control Script”.”

Customizing the Package Control Script

Check the definitions and declarations at the beginning of the control script using the information in the Package Configuration worksheet. You need to customize as follows:

Update the PATH statement to reflect any required paths needed to start your services.
If you are using LVM, enter the names of volume groups to be activated using the VG[] array parameters, and select the appropriate options for the storage activation command, including options for mounting and unmounting filesystems, if desired. Do not use the VXVM_DG[] or CVM_DG[] parameters for LVM volume groups.
If you are using CVM, enter the names of disk groups to be activated using the CVM_DG[] array parameters, and select the appropriate storage activation command, CVM_ACTIVATION_CMD. Do not use the VG[] or VXVM_DG[] parameters for CVM disk groups.
If you are using VxVM disk groups without CVM, enter the names of VxVM disk groups that will be imported using the VXVM_DG[] array parameters. Enter one disk group per array element. Do not use the CVM_DG[] or VG[] parameters for VxVM disk groups without CVM. Also, do not specify an activation command.
Add the names of logical volumes and file systems that will be mounted on them.
If you are using mirrored VxVM disks, specify the mirror recovery option VXVOL.
Select the appropriate options for the storage activation command (not applicable for basic VxVM disk groups), and also include options for mounting and unmounting filesystems, if desired.
Specify the filesystem mount retry and unmount count options.
Define IP subnet and IP address pairs for your package.
Add service name(s).
Add service command(s)
Add a service restart parameter, if desired.




	NOTE: Use care in defining service run commands. Each run command is executed by the control script in the following way: The cmrunserv command executes each run command and then monitors the process id of the process created by the run command. When the command started by cmrunserv exits, ServiceGuard determines that a failure has occurred and takes appropriate action, which may include transferring the package to an adoptive node. If a run command is a shell script that runs some other command and then exits, ServiceGuard will consider this normal exit as a failure. To avoid problems in the execution of control scripts, ensure that each run command is the name of an actual service and that its process remains alive until the actual service stops.

If you need to define a set of run and halt operations in addition to the defaults, create functions for them in the sections under the heading CUSTOMER DEFINED FUNCTIONS.

Optimizing for Large Numbers of Storage Units

A set of four variables is provided to allow performance improvement when employing a large number of filesystems or storage groups. For more detail, see the comments in the control script template. They are:

CONCURRENT_VGCHANGE_OPERATIONS—defines a number of parallel LVM volume group activations during package startup as well and deactivations during package shutdown.
CONCURRENT_FSCK_OPERATIONS—defines a number of parallel fsck operations that will be carried out at package startup.
CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS—defines a number of parallel mount operations during package startup and unmount operations during package shutdown.

Customizing the Control Script for RAC Instances

Use the package control script to perform the following:

Activation and deactivation of RAC volume groups.
Startup and shutdown of the RAC instance.
Monitoring of the RAC instance.

Set RAC environment variables in the package control script to define the correct execution environment for RAC.

Enter the names of the LVM volume groups you wish to activate in shared mode in the VG[] array. Use a different array element for each RAC volume group. (Remember that RAC volume groups must also be coded in the cluster configuration file using OPS_VOLUME_GROUP parameters.) Be sure to specify shared activation with the vgchange command by setting the VGCHANGE parameter as follows:

VGCHANGE="vgchange -a s”

If your disks are mirrored with LVM mirroring on separate physical paths and you want to override quorum, use the following setting:

VGCHANGE="vgchange -a s -q n”

Enter the names of the CVM disk groups you wish to activate in shared mode in the CVM_DG[] array. Use a different array element for each RAC disk group. (Remember that CVM disk groups must also be coded in the package ASCII configuration file using STORAGE_GROUP parameters.) Be sure to an appropriate type of shared activation with the CVM activation command. For example:

CVM_ACTIVATION_CMD="vxdg -g \$DiskGroup set activation=sharedwrite"

Do not define the RAC instance as a package service. Instead, include the commands that start up an RAC instance in the customer_defined_run_commands section of the package control script. Similarly, you should include the commands that halt an RAC instance in the customer_defined_halt_commands section of the package control script.

Define the Oracle monitoring command as a service command, or else use the special Oracle script provided with the ECM Toolkit.

Configuring an Oracle RAC Instance Package

To configure an Oracle RAC Instance Package, you can use either Serviceguard Manager or the command line.

The Enterprise Cluster Master Toolkit (ECMT) provides a generic Oracle database script template which can be used to configure package behavior for an Oracle RAC Instance package. This script is integrated by the user into the standard Serviceguard package control script, to simplify the startup, monitoring, and shutdown of an Oracle database instance. Refer to the ECMT documentation for details on the use of ECMT.

Each Oracle RAC database can have a database instance running on all nodes of a SGeRAC cluster. Therefore, it is not necessary to failover the database instance to a different SGeRAC node. This is the main difference between an Oracle RAC Instance Package and a single instance Oracle package.

Information for Creating the Oracle RAC Instance Package on a SGeRAC Node

Use the following steps to set up the pre-package configuration on a SGeRAC node:

Gather the RAC Instance SID_NAME. If you are using Serviceguard Manager, this is in the cluster Properties.
Example: SID_NAME=ORACLE_TEST0
For an ORACLE RAC Instance for a two node cluster, each node would have an SID_NAME.
Gather the RAC Instance package name for each node, which should be the same as the SID_NAME for each node
Example: ORACLE_TEST0
Gather the shared volume group name for the RAC database. In Serviceguard Manager, see cluster Properties.
Example: /dev/vgora92db
Create the Oracle RAC Instance Package directory

/etc/cmcluster/${SID_NAME}
Example: /etc/cmcluster/ORACLE_TEST0
If you are using the command line copy the Oracle shell script template from the ECMT default source directory to the package directory.

# cd /opt/cmcluster/toolkit/oracle
# cp oracle.sh /etc/cmcluster/${SID_NAME}/${SID_NAME}.sh
# chmod 755 /etc/cmcluster/${SID_NAME}/${SID_NAME}.sh
Example:
# cd /opt/cmcluster/toolkit/oracle
# cp oracle.sh /etc/cmcluster/ORACLE_TEST0/ORACLE_TEST0.sh
# chmod 755 /etc/cmcluster/ORACLE_TEST0/ORACLE_TEST0.sh
Gather the package service name for monitoring Oracle instance processes. In Serviceguard Manager, this information can be found under the “Services” tab. SERVICE_NAME[0]=${SID_NAME} SERVICE_CMD[0]=”etc/cmcluster/${SID_NAME}/${SID_NAME}.sh monitor”
SERVICE_RESTART[0]=”3”
Example: SERVICE_NAME[0]=ORACLE_TEST0 SERVICE_CMD[0]=”/etc/cmcluster/ORACLE_TEST0/ORACLE_TEST0.sh monitor”
SERVICE_RESTART[0]=”3”
Gather how to start the database using an ECMT script. In Serviceguard Manager, enter this filename for the control script start command.
/etc/cmcluster/${SID_NAME}/${SID_NAME}.sh start
Example: /etc/cmcluster/ORACLE_TEST0/ORACLE_TEST0.sh start
Gather how to stop the database using an ECMT script. In Serviceguard Manager, enter this filename for the control script start command.
/etc/cmcluster/${SID_NAME}/${SID_NAME}.sh halt
Example: /etc/cmcluster/ORACLE_TEST0/ORACLE_TEST0.sh halt

Using Serviceguard Manager to a Configure Oracle RAC Instance Package

The following steps uses the information from the example is section 2.2. Assume SGMgr has been started, the following. It is assumed that the SGeRAC cluster environment is configured and the ECMT can be used to start the Oracle RAC database instance.

Start Serviceguard Manager and Connect to cluster. Figure 2-1 shows a RAC Instance package for node “sg21”. The package name is ORACLE_TEST0
Figure 2-1 Serviceguard Manager display for a RAC Instance package
Create the Package.
Select the Parameters and select the parameters to edit.
Next select the check box “Enable template(x)” to enable Package Template for Oracle RAC. The template defaults can be reset with the “Reset template defaults” push button.
When enabling the template for Oracle RAC, the package can only be run on one node.
Select the Node tab and select the node to run this package.
Select Networks tab and add monitored subnet for package.
Select Services tab and configure services.
Select the Control Script tab and configure parameters.
Configure volume groups and customer defined run/halt functions.
Apply the package configuration after filling in the specified paramenters.

Enabling DB Provider Monitoring

To monitor a remote Serviceguard RAC cluster, the entry of the GUI user name and server name must be in /etc/cmcluster/cmclnodelist file on all nodes in the cluster to be viewed.

DB Provider Monitoring Using Control Access Policy (CAP)

To monitor a local SGeRAC cluster as a non-root user, the GUI user name, server name and user role (at least monitor role) must be configured through CAP.

For a remote cluster, any GUI user name (non-root or root), server name and user role (at least monitor role) should be configured through CAP.

Please refer Control Access Policy for Serviceguard Commands and API clients External Specification for details.