Managing the Cluster and Nodes

Managing the cluster involves the following tasks:

Starting the Cluster When All Nodes are Down
Adding Previously Configured Nodes to a Running Cluster
Removing Nodes from Operation in a Running Cluster
Halting the Entire Cluster

You can use ServiceGuard Manager, SAM or MC/ServiceGuard commands to start or stop the cluster, or to add or halt nodes. You can use SAM or commands to modify the cluster configuration. Starting the cluster means running the cluster daemon on one or more of the nodes in a cluster. You use different MC/ServiceGuard commands to start the cluster depending on whether all nodes are currently down (that is, no cluster daemons are running), or whether you are starting the cluster daemon on an individual node.During normal operation, the cluster functions continuously without intervention. When it becomes necessary to stop the entire cluster for such operations as replacing hardware or physically moving the nodes, you can manually halt the cluster and restart it at a later time. Stopping the cluster in this way has the effect of running the OPS halt scripts on each node to halt the operation of OPS and OPS applications as the cluster shuts down. Restarting the cluster has the effect of running the OPS start script on each node (if it is configured) to bring up Oracle Parallel Server and its applications.

Note the distinction that is made in this chapter between adding an already configured node to the cluster and adding a new node to the cluster configuration. An already configured node is one that is already entered in the cluster configuration file; a new node is added to the cluster by modifying the cluster configuration file.

Starting the Cluster When all Nodes are Down

You can use ServiceGuard Manager, SAM or MC/ServiceGuard commands to start the cluster. Starting or restarting the cluster has the effect of running the OPS instance package control script on each node (if one is configured) to bring up Oracle Parallel Server and its applications. If you are not running OPS in packages, you must activate storage and run OPS instances on each node separately after the cluster has formed.

Using ServiceGuard Manager to Start the Cluster

Select the cluster icon, then right-click to display the action menu. Select "Start the Cluster." The progress window shows messages as the action takes place. This will include messages for starting each node and package. Click OK on the progress window when the operation is complete.

Using SAM to Start the Cluster

In SAM, choose Clusters, then High Availability Clusters. In the Cluster Administration area, highlight the cluster you wish to start, then select "Start Cluster" from the Actions menu. Indicate whether you wish to bring up the cluster on all nodes or on a subset of nodes. If you choose a subset, then you must select the nodes from the list. This option should be used only when you are sure that the cluster is not already running on any node.

At the verification prompt, select OK to start the cluster.

Using MC/ServiceGuard Commands to Start the Cluster

Use the cmruncl command to start the cluster when all cluster nodes are down. Use the -n option to specify a particular group of nodes. Without this option, all nodes will be started. The following starts all nodes configured in the cluster:

# cmruncl -v

The following example starts up the locally configured cluster only on ftsys9 and ftsys10. (This form of the command should only be used when you are sure that the cluster is not already running on any node.)

# cmruncl -v -n ftsys9 -n ftsys10




	WARNING! MC/ServiceGuard cannot guarantee data integrity if you try to start a cluster with the cmruncl -n command while a subset of the cluster's nodes are already running a cluster. If the network connection is down between nodes, using cmruncl -n might result in a second cluster forming, and this second cluster might start up the same applications that are already running on the other cluster. The result could be two applications overwriting each other's data on the disks.

Adding Previously Configured Nodes to a Running Cluster

You can use ServiceGuard Manager, SAM or MC/ServiceGuard commands to bring a configured node up within a running cluster.




	NOTE: If you have configured your OPS instances to run as a package, then they will activate storage and start automatically after the node joins the cluster. If you are not using OPS instance packages, then you must activate storage and start OPS separately once the node joins the cluster.

Using ServiceGuard Manager to Add a Node to the Cluster

Select the node icon, then right-click to display the action menu. Select "Start the Node." The progress window shows messages as the action takes place. This will include starting any packages that are eligible to run on the node. Click OK on the progress window when the operation is complete.

Using SAM to Add Previously Configured Nodes to a Running Cluster

In the Cluster Administration area, highlight the cluster you are adding the node to, then select "Specify Nodes to Join the Cluster." Choose the node or nodes you are adding, then choose OK. Only nodes that are members of the current cluster configuration may be selected.

Using MC/ServiceGuard Commands to Add Previously Configured Nodes to a Running Cluster

Use the cmrunnode command to add one or more nodes to an already running cluster. Any node you add must already be a part of the cluster configuration. The following example adds node ftsys8 to the cluster that was just started with only nodes ftsys9 and ftsys10:

# cmrunnode -v ftsys8

Since the node's cluster is already running, the node joins the cluster and packages may be started. If the node does not find its cluster running, or the node is not part of the cluster configuration, the command fails.

Removing Nodes from Operation in a Running Cluster

You can use SAM or HP-UX commands to remove nodes from operation in a cluster. This operation removes the node from cluster operation by halting the cluster daemon, but it does not modify the cluster configuration. To remove a node from the cluster configuration permanently, you must recreate the cluster configuration file. See the next section.




	NOTE: If you have configured your OPS instances to run as a package, then they will halt automatically as a part of shutting down the node. If not, then you must halt any OPS instances on the node and deactivate storage separately before the node leaves the cluster.

Using ServiceGuard Manager to Remove a Node from the Cluster

Select the node icon, then right-click to display the action menu. Select "Halt the Node." The progress window shows messages as the action takes place. This will include moving any packages on the node to adoptive nodes, if appropriate. Click OK on the progress window when the operation is complete.

Using SAM to Remove Nodes from Operation

In the Cluster Administration area, highlight the cluster you are interested in, then select "Specify Nodes to Leave the Cluster" from the Actions menu. Choose the node or nodes, then select OK.

If a package is running on a node you are shutting down, you will see a message asking for a verification that you wish to shut down the node anyway. Reply OK to verify shutdown. Following shutdown, any package that can switch to an adoptive node will start up on the adoptive node.

Using MC/ServiceGuard Commands to Remove Nodes from Operation

Use the cmhaltnode command to halt one or more nodes in a cluster. The cluster daemon on the specified node stops, and the node is removed from active participation in the cluster.

To halt a node with a running package, use the -f option. If a package was running that can be switched to an adoptive node, the switch takes place and the package starts on the adoptive node. For example, the following command causes the ServiceGuard daemon running on node ftsys9 in the sample configuration to halt and the package running on ftsys9 to move to ftsys10:

# cmhaltnode -f -v ftsys9

This halts any packages running on the node ftsys9 by executing the halt instructions in each package's control script. ftsys9 is halted and the packages start on the adoptive node, ftsys10.

The use of cmhaltnode is a convenient way of bringing a node down for system maintenance while keeping its packages available on other nodes. After maintenance, the package can be returned to its primary node. See "Moving a Package," below.

To return a node to the cluster, use cmrunnode.




	NOTE: It is recommended to run cmhaltnode prior to running the HP-UX shutdown command, especially for cases where a packaged application might have trouble during shutdown and not halt cleanly.

Halting the Entire Cluster

You can use ServiceGuard Manager, SAM or MC/ServiceGuard commands to halt a running cluster. If you are not running OPS in packages, you must halt OPS instances and deactivate storage on each node separately before you halt the cluster.

Using ServiceGuard Manager to Halt the Cluster

Select the cluster, then right-click to display the action menu. Select "Halt the Cluster." The progress window shows messages as the action takes place. This will include messages for halting each package and node. Click OK on the progress window when the operation is complete.

Using SAM to Halt the Cluster

In the Cluster Administration area, highlight the cluster you wish to halt, then select "Shut Down Cluster" from the Actions menu to halt the selected cluster on all nodes. At the verification prompt, choose OK.

Using MC/ServiceGuard Commands to Halt a Cluster

The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their MC/ServiceGuard daemons. You can use the -f option to force the cluster to halt even when packages are running. This command can be issued from any running node. Example:

# cmhaltcl -f -v

This halts all nodes that are configured in the cluster.