Responses to Failures

Transfer of Control (TOC) When a Node Fails

The most dramatic response to a failure in an MC/LockManager cluster is an HP-UX TOC (Transfer of Control), which is an immediate halt of the SPU without a graceful shutdown. This TOC is done to protect the integrity of your data.

A TOC is done if there is a kernel hang, a kernel spin, a runaway real-time process, or if the MC/LockManager daemon, cmcld, fails. A TOC also takes place if the GMS process dies (OPS 8.0.5 and later). During this event, a system dump is performed and the following message is sent to the console:

MC/LockManager: Unable to maintain contact with cmcld daemon.
Performing TOC to ensure data integrity.

A TOC is also initiated by MC/LockManager itself under specific circumstances. If the service failfast parameter is enabled in the package configuration file, the entire node will fail with a TOC whenever there is a failure of that specific service. If the package failfast parameter is enabled in the package configuration file, the entire node will fail with a TOC whenever there is a failure causing the package control script to exit. In addition, a node-level failure may also be caused by events independent of a package and its services. Loss of the heartbeat or loss of the MC/LockManager or other critical daemons will cause a node to fail even when its packages and their services are functioning.

Responses to Hardware Failures

If a serious system problem occurs, such as a panic or physical disruption of the SPU's circuits, MC/LockManager recognizes a node failure and transfers the packages currently running on that node to an adoptive node elsewhere in the cluster. The new location for each package is determined by that package's configuration file, which lists primary and alternate nodes for the package. Transfer of a package to another node does not transfer the program counter. Processes in a transferred package will restart from the beginning. In order for an application to be expeditiously restarted after a failure, it must be "crash-tolerant"; that is, all processes in the package must be written so that they can detect such a restart. This is the same application design required for restart after a normal system crash.

In the event of a LAN interface failure, a local switch is done to a standby LAN interface if one exists; otherwise, the node fails with a TOC.

MC/LockManager does not respond to disk failure; disk protection is provided by the separate product MirrorDisk/UX (root disks or non-OPS data only) or HP High Availability Disk Arrays. Packages do not switch as a result of disk failures.

MC/LockManager does not respond directly to power failures, although a loss of power to an individual cluster component may appear to MC/LockManager like the failure of that component, and will result in the appropriate switching behavior.

Power protection is provided by PowerTrust, HP's uninterruptible power supply.

Responses to Package and Service Failures

In the default case, the failure of the package or of a service within a package causes the package to shut down by running the control script with the 'stop' parameter, and then restarting the package on an alternate node. If the package manager receives a report of an EMS monitor event showing that a resource is down, the package fails.

If you wish, you can modify this default behavior by specifying that the node should crash (TOC) before the transfer takes place. In cases where package shutdown may take a long time but the package is crash-tolerant and can recover quickly on restart, this option can make the package and its associated applications available to users more quickly. Remember, however, that when the node crashes, all packages on the node are halted abruptly.

The settings of package and service failfast parameters during package configuration will determine the exact behavior of the package and the node in the event of failure. The section on "Package Configuration Parameters" in the "Planning" chapter contains details on how to choose an appropriate failover behavior.

Service Restarts

You can allow a service to restart locally following a failure. To do this, you indicate a number of restarts for each service in the package control script. When a service starts, the variable RESTART_COUNT is set in the service's environment. The service, as it executes, can examine this variable to see whether it has been restarted after a failure, and if so, it can take appropriate action such as cleanup.

Network Communication Failure

An important element in the cluster is the health of the network itself. As it continuously monitors the cluster, each node listens for heartbeat messages from the other nodes confirming that all nodes are able to communicate with each other. If a node does not hear these messages within the configured amount of time, a node timeout occurs, resulting in a TOC. In a two-node cluster, the use of an RS-232 line prevents a TOC from the momentary loss of heartbeat on the LAN due to network saturation. The RS232 line also assists in quickly detecting network failures when they occur.