Configuring the Cluster

This section describes how to define the basic cluster configuration. To do this in SAM, read the next section. If you want to use MC/LockManager commands, skip ahead to the section entitled "Using MC/LockManager Commands to Configure the Cluster."

Using SAM to Configure the Cluster

To configure a high availability cluster, use the following steps on the configuration node (ftsys9):

In SAM, select Clusters, then the High Availability Clusters option.
Choose the Cluster Configuration option. SAM displays a Cluster Configuration screen. If no clusters have yet been configured, the list area will be empty. If there are one or more HA clusters already configured on your local network, you will see them listed.
Select the Actions menu, and choose Create Cluster Configuration. A step menu appears.
Choose each required step in sequence, filling in the dialog boxes with required information, or accepting the default values shown. For information about each step, choose Help.
When finished with all steps, select at the Step Menu screen. This action creates the cluster configuration file and then copies the file to all the nodes in the cluster. When the file copying is finished, you return to the Cluster Configuration screen.
Exit from the Cluster Configuration screen, returning to the High Availability Clusters menu.




	NOTE: In addition to creating and distributing a binary cluster configuration file, SAM creates an ASCII cluster configuration file, named /etc/cmcluster/cmclconfig.ascii. This file is available as a record of the choices entered in SAM.

Skip ahead to the section "Configuring the Distributed Lock Manager."

Using MC/LockManager Commands to Configure the Cluster

Use the cmquerycl command to specify a set of nodes to be and to generate a template for the cluster configuration file. Here is an example of the command as issued from node ftsys9:

# cmquerycl -v -C /etc/cmcluster/cmclconf.ascii \ -n ftsys9 -n ftsys10

The example creates an ASCII template file in the default cluster configuration directory, /etc/cmcluster. The ASCII file is partially filled in with the names and characteristics of cluster components on the two nodes ftsys9 and ftsys10. Edit the filled-in cluster characteristics as needed to define the desired cluster. It is strongly recommended that you edit the file to send heartbeat over all possible networks, as shown in the following example.

Cluster Configuration Template File

# **********************************************************************
# ********* HIGH AVAILABILITY CLUSTER CONFIGURATION FILE ***************
# ***** For complete details about cluster parameters and how to    ****
# ***** set them, consult the cmquerycl(1m) manpage or your manual. ****
# **********************************************************************
 
# Enter a name for this cluster.  This name will be used to identify the
# cluster when viewing or manipulating it.
 
CLUSTER_NAME            cluster1
 
 
# Cluster Lock Device Parameters.  This is the volume group that
# holds the cluster lock which is used to break a cluster formation
# tie.  This volume group should not be used by any other cluster
# as cluster lock device.
 
FIRST_CLUSTER_LOCK_VG   /dev/vg01
 
 
# Definition of nodes in the cluster.
# Repeat node definitions as necessary for additional nodes.
 
NODE_NAME               ftsys9
  NETWORK_INTERFACE     lan0
    HEARTBEAT_IP        15.13.171.32
  NETWORK_INTERFACE     lan3
    HEARTBEAT_IP        192.6.7.3
  NETWORK_INTERFACE     lan4
  NETWORK_INTERFACE     lan1
    HEARTBEAT_IP        192.6.143.10
  FIRST_CLUSTER_LOCK_PV /dev/dsk/c1t2d0
 
 
# List of serial device file names
# For example:
# SERIAL_DEVICE_FILE    /dev/tty0p0
 
 
# Primary Network Interfaces on Bridged Net 1: lan0.
#   Warning: There are no standby network interfaces on bridged net 1.
# Primary Network Interfaces on Bridged Net 2: lan3.
#   Possible standby Network Interfaces on Bridged Net 2: lan4.
# Primary Network Interfaces on Bridged Net 3: lan1.
#   Warning: There are no standby network interfaces on bridged net 3.
 
NODE_NAME               ftsys10
  NETWORK_INTERFACE     lan0
    HEARTBEAT_IP        15.13.171.30
  NETWORK_INTERFACE     lan3
    HEARTBEAT_IP        192.6.7.4
  NETWORK_INTERFACE     lan4
  NETWORK_INTERFACE lan1
    HEARTBEAT_IP        192.6.143.20
  FIRST_CLUSTER_LOCK_PV /dev/dsk/c1t2d0
 
 
# List of serial device file names
# For example:
# SERIAL_DEVICE_FILE    /dev/tty0p0
 
 
# Primary Network Interfaces on Bridged Net 1: lan0.
#   Warning: There are no standby network interfaces on bridged net 1.
# Primary Network Interfaces on Bridged Net 2: lan3.
#   Possible standby Network Interfaces on Bridged Net 2: lan4.
# Primary Network Interfaces on Bridged Net 3: lan1.
#   Warning: There are no standby network interfaces on bridged net 3.
 
 
# Cluster Timing Parameters (microseconds).
 
HEARTBEAT_INTERVAL       1000000
NODE_TIMEOUT             2000000
 
 
# Configuration/Reconfiguration Timing Parameters (microseconds).
 
AUTO_START_TIMEOUT       600000000
NETWORK_POLLING_INTERVAL 2000000
 
# Package Configuration Parameters.
# Enter the maximum number of packages which will be configured in the cluster.
# You can not add packages beyond this limit.
# This parameter is required.
MAX_CONFIGURED_PACKAGES            10
 
#
# List of cluster aware Volume Groups. These volume groups
# will be used by clustered applications via the vgchange -a e command.
# For example: 
# VOLUME_GROUP          /dev/vgdatabase
# VOLUME_GROUP          /dev/vg02
VOLUME_GROUP            /dev/vg01
VOLUME_GROUP            /dev/vg02
 
# List of OPS Volume Groups.# Formerly known as DLM Volume Groups, these volume groups
# will be used by OPS cluster applications via
# the vgchange -a s command. (Note: the name DLM_VOLUME_GROUP
# is also still supported for compatibility with earlier versions.)
# For example:# OPS_VOLUME_GROUP              /dev/vgdatabase.
# OPS_VOLUME_GROUP              /dev/vg02.
OPS_VOLUME_GROUP /dev/vg_ops
 
# DLM parameters.
 
DLM_ENABLED   NO
DLM_CONNECT_TIMEOUT  30000000
DLM_PING_INTERVAL  20000000
DLM_PING_TIMEOUT  60000000
DLM_RECONFIG_TIMEOUT  120000000
DLM_COMMFAIL_TIMEOUT  150000000
DLM_HALT_TIMEOUT  240000000
 
 
# GMS parameters.
# When using Oracle Parallel Server version 8.0 or later, set
# DLM_ENABLED to NO above, set GMS_ENABLED to YES below, and
# enter values for the other parameters as specified in your
# Oracle documentation.
 
GMS_ENABLED             NO
GMS_CONNECT_TIMEOUT     30000000
GMS_LOCATION

The man page for the cmquerycl command lists the definitions of all the parameters that appear in this file. Many are also described in the "Planning" chapter. Modify the /etc/cmcluster/cmclconf.ascii file to your requirements, using the data on the cluster configuration worksheet.

In the file, keywords are separated from definitions by white space. Comments are permitted, and must be preceded by a pound sign (#) in the far left column. See the man page for the cmquerycl command for more details.

Identifying Volume Groups for Packages

The file will include entries for all package volume groups that are to be defined as cluster-aware, that is, those which can be accessed by packages running on different nodes in the cluster at different times. A separate VOLUME_GROUP line should appear for each volume group that will be activated by any package running in the cluster. To leave a volume group unmarked, remove the volume group name from the ASCII file.




	NOTE: If a volume group is not cluster-aware, then it cannot be activated by a package control script.

Identifying OPS Volume Groups

The template file will also include entries for all volume groups used by the Oracle Parallel Server that are accessed concurrently by the different nodes in the cluster. These volume groups are activated by the vgchange -a s command in the control script that activates each OPS instance. A separate OPS_VOLUME_GROUP line should appear for each volume group that will be activated in shared mode. Volume groups that will be used by Oracle Parallel Server must be labelled OPS_VOLUME_GROUP.




	NOTE: It's important that only volume groups used by OPS be listed with the `OPS_VOLUME_GROUP` parameter, since these volume groups will be marked for activation in shared mode. Volume groups used by other packages should be listed with the `VOLUME_GROUP` parameter described above. You may need to change the default assignments in order to get this correct.

Redeploying Previously Configured Volume Groups

In configuring a new cluster, if you are using volume groups that were used in a previous cluster configuration, you should ensure that they are not currently cluster aware (marked with a cluster id). You can use the following command to remove the cluster id if necessary:

# vgchange -c n

Identifying the Cluster Lock Volume Group and Disk

A cluster lock disk is required for two node clusters like the one in this example. The disk must be accessible to all nodes and must be powered separately from the nodes. Refer to the section "Use of the Cluster Lock" in Chapter 3 for additional information.

The default FIRST_CLUSTER_LOCK_VG and FIRST_CLUSTER_LOCK_PV supplied in the ASCII template created with cmquerycl are the volume group and physical volume name of a disk chosen based on minimum failover time calculations. You should ensure that this disk meets your power wiring requirements. If necessary, choose a disk powered by a circuit which powers fewer than half the nodes in the cluster.

If necessary, you can configure a second cluster lock. Enter the following parameters in the cluster configuration file:

SECOND_CLUSTER_LOCK_VG /dev/volume-group
SECOND_CLUSTER_LOCK_PV /dev/dsk/block-special-file

where the /dev/volume-group is the name of the second volume group and block-special-file is the physical volume name of a lock disk in the chosen volume group. These lines should be added to the information for each node.

To display the failover times of disks, use the cmquerycl command, specifying all the nodes in the cluster:

# cmquerycl -v -n ftsys9 -n ftsys10

The output of the command lists the disks connected to each node together with the re-formation time associated with each.

Specifying Maximum Number of Configured Packages

MC/LockManager preallocates memory and threads at cluster startup time. It calculates these values based on the number of packages specified in the MAX_CONFIGURED_PACKAGES parameter in the cluster configuration file. This value must be equal to or greater than the number of packages currently configured in the cluster. The default is 0, which means that you must enter a value if you wish to use packages. The absolute maximum number of packages per cluster is 30.

Identifying Serial Heartbeat Connections

If you are using a serial (RS232) line as a heartbeat connection, use the SERIAL_DEVICE_FILE parameter and enter the device file name that corresponds to the serial port you are using on each node. Be sure that the serial cable is securely attached during and after configuration.

Enabling DLM (OPS 7.3.x)

When the DLM_ENABLED parameter is set to YES, the DLM is started in the cluster when the cluster starts or reboots. If DLM has not yet been configured, and this parameter is set to YES, the cluster will not start successfully. If you set DLM_ENABLED to YES, you must set GMS_ENABLED to NO.

Enabling GMS (OPS 8.0.5)

When the GMS_ENABLED parameter is set to YES, the OPS instances access the cluster's group membership services. GMS is available whenever the cluster is running. If you set GMS_ENABLED to YES, you must set DLM_ENABLED to NO.