Designing a Disaster Tolerant Architecture for use with MetroCluster/CA

Although you can use one arbitrator, having two arbitrators provides greater flexibility in taking systems down for planned outages as well as providing better protection against multiple points of failure. Using two arbitrators:

If you use a single arbitrator system, special procedures must be followed during planned downtime to remain protected. Systems must be taken down in pairs, one from each of the data centers, so that the MC/ServiceGuard quorum is maintained after a node failure. If the arbitrator itself must be taken down, disaster recovery capability is at risk if one of the other systems fails.

Arbitrator systems can be used to perform important and useful work such as:

Each XP Series disk array must be configured with redundant CA links, each of which is connected to a different LCP or RCP card. To prevent a single point of failure (SPOF), there must be at least two physical boards in each XP for the CA links. Each board usually has multiple ports. However, a redundant CA link must be connected to a port on a different physical board from the board that has the primary CA link. When using bi-directional configurations, where data center A backs up data center B and data center B backs up data center A, you must have at least four CA links, two in each direction. Four CA links are also required in uni-directional configurations in which you want to allow failback.

When a cluster initially forms, all systems must be available to form the cluster (100% Quorum requirement).

A quorum is dynamic and is recomputed after each system failure. For instance, if you start out with an 8-node cluster and two systems fail, that leaves 6 out 8 surviving nodes, or a 75% quorum. The cluster size is reset to 6 nodes. If two more nodes fail, leaving 4 out of 6, quorum is 67%.

Each time a cluster forms, there must be more than 50% quorum to reform the cluster. A cluster lock disk is normally used as the tie-breaker when quorum is exactly 50%. However, a cluster lock disk is not supported with MetroCluster with Continuous Access XP. Therefore, a quorum of 50% or less will cause the remaining nodes to halt.

Taking a node off-line for planned maintenance is treated the same as a node failure in these scenarios. Study these scenarios to make sure you do not put your cluster at risk during planned maintenance.

Figure 3-2 Failover Scenario with a Single Arbitrator

The scenarios in Table 3-3 “Node Failure Scenarios with One Arbitrator Fm Variable:Table Continuation”, based on Figure 3-2 “Failover Scenario with a Single Arbitrator”, illustrate possible results if one or more nodes fail in a configuration with a single arbitrator.

* Cluster can be manually started with the remaining node.

Table 3-4 “Data Center Failure Scenarios with One Arbitrator” illustrates possible results if a data center fails in a configuration with a single arbitrator.

* Cluster can be manually started with the remaining node.

With a single arbitrator node, the cluster is at risk each time a node fails or comes down for planned maintenance.

Having two arbitrator nodes adds extra protection during node failures and allows you to do planned maintenance on arbitrator nodes without losing the cluster should a disaster occur.

Figure 3-3 Failover Scenario with Two Arbitrators

The scenarios in Table 3-5 “Failure Scenarios with Two Arbitrators” illustrate possible results if a data center or one or more nodes fail in a configuration with two arbitrators. Note that 3 of the 4 scenarios that caused a cluster halt with a single arbitrator, do not cause a cluster halt with two arbitrators.

* Cluster can be manually started with the remaining node.