Reconfiguring a Cluster

Reconfiguring a Running Cluster

You can add new nodes to the cluster configuration or delete nodes from the cluster configuration while the cluster is up and running provided you are using VERITAS CVM storage. Note the following, however:

You cannot change the lock disk configuration while the cluster is running.
You cannot remove an active node from the cluster. You must halt the node first.
You cannot delete an active storage group from the cluster configuration. You must halt any package that uses the storage group and ensure that the volume is inactive before deleting it.
You cannot change cluster timing parameters.
The only configuration change allowed while a node is unreachable (for example, completely disconnected from the network) is to delete the unreachable node from the cluster configuration. If there are also packages that depend upon that node, the package configuration must also be modified to delete the node. This all must be done in one configuration request (cmapplyconf command).

Changes to the package configuration are described in a later section.

Table 7-2 Types of Changes to Permanent Cluster Configuration

Change to the Cluster Configuration	Required Cluster State
Add a new node	All cluster nodes must be running. Serial heartbeat must not be configured.
Delete a node	A node can be deleted even though it is unavailable or unreachable.
Add a storage group	Cluster may be running or halted. Must be running for CVM disk groups.
Delete a storage group	Cluster may be running or halted. Must be running for CVM disk groups. Packages that use the storage group will not be able to start again until their control scripts are modified.
Change Maximum Configured Packages	Cluster must not be running.
Change Timing Parameters	Cluster must not be running.
Change Cluster Lock Configuration	Cluster must not be running.
Change serial device files	Cluster must not be running.
Change IP addresses for heartbeats or monitored subnets	Cluster must not be running.

The following sections describe how to perform dynamic reconfiguration tasks using SAM or using MC/ServiceGuard commands.

Using SAM to Add Nodes to the Configuration While the Cluster is Running

Use the Cluster Configuration area of SAM, and choose "Modify Cluster Configuration" from the Actions menu. Then select "Modify Cluster Name and Nodes." Highlight the name of the node you wish to add to the cluster in the list on the right of the screen, then click the Add button. When you are finished changing the node list, click OK.

Using MC/ServiceGuard Commands to Add Nodes to the Configuration While the Cluster is Running

Use the following procedure to add a node with HP-UX commands. For this example, nodes ftsys8 and ftsys9 are already configured in a running cluster named cluster1, and you are adding node ftsys10.

Use the following command to store a current copy of the existing cluster configuration in a temporary file:
# cmgetconf -C temp.ascii
Specify a new set of nodes to be configured and generate a template of the new configuration:
# cmquerycl -C clconfig.ascii -c cluster1 \ -n ftsys8 -n ftsys9 -n ftsys10
Edit the file clconfig.ascii to check the information about the new node.
Verify the new configuration:
# cmcheckconf -C clconfig.ascii
Apply the changes to the configuration and send the new binary configuration file to all cluster nodes:
# cmapplyconf -C clconfig.ascii

Use cmrunnode to start the new node, and, if desired, set the AUTOSTART_CMCLD parameter to 1 in the /etc/rc.config.d/cmcluster file to enable the new node to join the cluster automatically each time it reboots.

Using SAM to Delete Nodes from the Configuration While the Cluster is Running

In the Cluster Administration area of SAM, halt the node that you wish to remove from the cluster configuration. Then go to the Cluster Configuration area, and choose "Modify Cluster Configuration" from the Actions menu. Then select "Modify Cluster Name and Nodes." Highlight the name of the node you wish to delete from the cluster in the list of configured nodes, then click the Remove button. When you are finished changing the node list, click OK.

If the node you wish to delete is unreachable (disconnected from the LAN, for example), you can use SAM to delete the node only if there are no packages which specify the unreachable node. If there are packages that depend on the unreachable node, use MC/ServiceGuard commands as described in the next section.

Using MC/ServiceGuard Commands to Delete Nodes from the Configuration While the Cluster is Running

Use the following procedure to delete a node with HP-UX commands. For this example, nodes ftsys8, ftsys9 and ftsys10 are already configured in a running cluster named cluster1, and you are deleting node ftsys10.

Use the following command to store a current copy of the existing cluster configuration in a temporary file:
# cmgetconf -C temp.ascii
Specify the new set of nodes to be configured (omitting ftsys10) and generate a template of the new configuration:
# cmquerycl -C clconfig.ascii -c cluster1 -n ftsys8 -n ftsys9
Edit the file clconfig.ascii to check the information about the nodes that remain in the cluster.
Verify the new configuration:
# cmcheckconf -C clconfig.ascii
Apply the changes to the configuration and send the new binary configuration file to all cluster nodes:
# cmapplyconf -C clconfig.ascii

NOTE: If you are attempting to remove an unreachable node that has many packages dependent on it, especially if the dependent packages use a large number of EMS resources, you may see the following message:

The configuration change is too large to process while the cluster is running. 
Split the configuration change into multiple requests or halt the cluster.

In this situation, you must halt the cluster to remove the unreachable node.

Changing the VxVM or CVM Storage Configuration

You can add VxVM disk groups to the cluster configuration while the cluster is running. To add new CVM disk groups, the cluster must be running.

If you are creating new CVM disk groups, be sure to determine the master node on which to do the creation by using the following command:

# vxdctl -c mode

One node will identify itself as the master. Create disk groups from this node. You must add the storage group to the cluster ASCII file in a STORAGE_GROUP statement, then issue the cmapplyconf command.

Similarly, you can delete VxVM or CVM disk groups provided they are not being used by a cluster node at the time.




	NOTE: If you are removing a disk group from the cluster configuration, make sure that you also modify or delete any package control script that imports and deports this disk group. If you are removing a CVM disk group, be sure to remove the `STORAGE_GROUP` entries for the disk group from the package ASCII file.

Reconfiguring a Halted Cluster

If your network or LAN card configuration changes on the cluster, you may need to modify the basic cluster configuration. This can be done through SAM or with HP-UX commands. The basic process is as follows:

Halt packages.
Ensure that the OPS database is not active on either node.
Deactivate and unshare any shared volume groups.
Halt the cluster.
Change the LAN configuration as needed.
Using either SAM high availability options or HP-UX commands, create a new configuration file and propagate the new configuration to all the nodes in the cluster.
Start up the cluster to see if it forms successfully.
Reboot all nodes. The cluster should reform, and the OPS instances and packages should come up again.

When reconfiguring the cluster, use the following guidelines.

Before executing cmapplyconf, mark the cluster lock volume group non-shareable if the volume group is being used as a shared database volume group:
# vgchange -S n -c n VGName
Then activate it in normal mode:
# vgchange -a y VGName
You may now execute the cmapplyconf command.
Before running cmruncl or cmrunnode on the new cluster configuration, deactivate the cluster lock volume group:
# vgchange -a n VGName
After bringing the cluster up with cmrunnode or cmruncl, mark each database volume group (including the cluster lock volume group, if it is being used for OPS data) with the new cluster ID and make it shareable:
# vgchange -S y -c y VGName
Now the shared volume groups can be activated in shared mode:
# vgchange -a s -p VGName

For more details about creating the configuration file, see "“Configuring the Cluster ”" in the chapter "Chapter 5 “Building an OPS Cluster Configuration”."

Changing Lock Volume Group Configuration

If you decide to change the lock volume group in your configuration, you must issue the following commands on the lock volume group before reconfiguration will succeed:

# vgchange -S n -c n vg_ops 
# vgchange -a y vg_ops

This process is needed when you modify the lock volume group configuration, although you can use the commands to convert any shared, cluster-bound ServiceGuard volume group into a standard (non-shareable) LVM volume group.