Volume Managers for Data Storage

A volume manager is a tool that lets you create units of disk storage known as storage groups. Storage groups contain logical volumes for use on single systems and in high availability clusters. In ServiceGuard OPS Edition clusters, storage groups are activated by package control scripts.

Types of Redundant Storage

In ServiceGuard OPS Edition, there are two types of supported shared data storage: mirrored individual disks (also known as JBODs, for "just a bunch of disks"), and external disk arrays which configure redundant storage in hardware. Mirroring is known as RAID1, while the redundancy available in a disk array is known as RAID5. Here are some differences between the two storage methods:

If you are using JBODs, the basic element of storage is an individual disk. This disk must be paired with another disk to create a mirror (RAID1). (ServiceGuard configurations usually have separate mirrors on different storage devices).
If you have a disk array, the basic element of storage is a LUN, which already provides storage redundancy via RAID5.

Examples of Mirrored Storage

Figure 3-20 “Physical Disks Within Shared Storage Units” shows an illustration of mirrored storage using HA storage racks. In the example, node1 and node2 are cabled in a parallel configuration, each with redundant paths to two shared storage devices. Each of two nodes also has two (non-shared) internal disks which are used for the root file system, swap etc. Each shared storage unit has three disks, The device file names of the three disks on one of the two storage units are c0t0d0, c0t1d0, and c0t2d0. On the other, they are c1t0d0, c1t1d0, and c1t2d0.

Figure 3-20 Physical Disks Within Shared Storage Units

Figure 3-21 “Mirrored Physical Disks” shows the individual disks combined in a multiple disk mirrored configuration.

Figure 3-21 Mirrored Physical Disks

Figure 3-22 “Multiple Devices Configured in Volume Groups” shows the mirrors configured into LVM volume groups, shown in the figure as /dev/vgpkgA and /dev/vgpkgB. The volume groups are activated by ServiceGuard packages for use by highly available applications.

Figure 3-22 Multiple Devices Configured in Volume Groups

Examples of Storage on Disk Arrays

Figure 3-23 “Physical Disks Combined into LUNs” shows an illustration of storage configured on a disk array. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system.

Figure 3-23 Physical Disks Combined into LUNs




	NOTE: LUN definition is normally done using utility programs provided by the disk array manufacturer. Since arrays vary considerably, you should refer to the documentation that accompanies your storage unit.

Figure 3-24 “Multiple Paths to LUNs” shows LUNs configured with multiple paths (links) to provide redundant pathways to the data.

Figure 3-24 Multiple Paths to LUNs

Finally, the multiple paths are configured into volume groups as shown in Figure 3-25 “Multiple Paths in Volume Groups”.

Figure 3-25 Multiple Paths in Volume Groups

Types of Volume Manager

ServiceGuard OPS Edition allows a choice of three volume managers for data storage:

HP-UX Logical Volume Manager (LVM) and (optionally) MirrorDisk/UX
VERITAS Volume Manager (VxVM)—Base and Add-on Products
VERITAS Cluster Volume Manager (CVM)

Separate sections in Chapters 5 and 6 explain how to configure cluster storage using all of these volume managers. The rest of the present section explains some of the differences among these available volume managers and offers suggestions about appropriate choices for your cluster environment.




	NOTE: The HP-UX Logical Volume Manager is described in Managing Systems and Workgroups. A complete description of VERITAS volume management products is available in the VERITAS Volume Manager for HP-UX Release Notes.

HP-UX Logical Volume Manager

Logical Volume Manager (LVM) is the legacy storage management product on HP-UX. Included with the operating system, LVM is available on all cluster nodes. It supports the use of MirrorDisk/UX, which is an add-on product that allows disk mirroring with up to two mirrors (for a total of three copies of the data). Currently, the root disk must be configured as an LVM volume group. (Note that the HP-UX root disk is not the same as the VERITAS root disk group, which must be configured in addition to the HP-UX root disk on any node that uses VERITAS volume manager products.) The ServiceGuard cluster lock disk also is configured using a disk configured in an LVM volume group.

LVM continues to be supported on HP-UX single systems and on ServiceGuard clusters.

SLVM

ServiceGuard OPS Edition is shipped with the HP-UX Shared Logical Volume Manager (SLVM), which allows you to activate LVM volume groups for shared write access on multiple nodes configured as an OPS cluster. SLVM continues to be supported on HP-UX systems with ServiceGuard OPS Edition installed.

VERITAS Volume Manager (VxVM)

The Base VERITAS Volume Manager for HP-UX (Base-VXVM) is provided at no additional cost with HP-UX 11i. This includes basic volume manager features, including a Java-based GUI. It is possible to configure cluster storage for ServiceGuard with only Base-VXVM. However, only a limited set of features is available.

The add-on product, VERITAS Volume Manager for HP-UX (B9116AA), provides a full set of enhanced volume manager capabilities in addition to basic volume management. This includes features such as mirroring, dynamic multipathing for active/active storage devices, and hot relocation. The VERITAS FastResync option for HP-UX (B9118AA) is another add-on product, available with an additional charge, that reduces the time taken to resynchronize a split mirror to the volume. This product requires the purchase of the VERITAS Volume Manager for HP-UX (B9116AA).




	NOTE: VxVM disk groups may not be used for shared read/write access to OPS data. Shared read/write access for OPS requires SLVM volume groups or CVM disk groups.

VxVM can be used in clusters that:

are of any size, up to 16 nodes.
require a fast cluster startup time.
do not require shared storage group activation.
do not have all nodes cabled to all disks.
need to use RAID 5 or striped mirroring.

Propagation of Disk Groups in VxVM

With VxVM, a disk group can be created on any node, with the cluster up or not. The user then needs to go to each node and validate that disk group by trying to import it. Thus, although there are more steps required for propagating disk groups with VxVM than with CVM, you have the freedom to create the disk group from any node.

Package Startup Time with VxVM

With VxVM, each disk group is imported by the package control script that uses the disk group. This means that cluster startup time is not affected, but individual package startup time might be increased because VxVM imports the disk group at the time the package starts up.

VERITAS Cluster Volume Manager (CVM)

You may choose to configure cluster storage with the VERITAS Cluster Volume Manager for HP-UX (CVM) instead of the Volume Manager (VxVM). Base-VXVM provides some basic cluster features when ServiceGuard OPS Edition is installed; up to two OPS cluster nodes are supported, but there is no support for software mirroring, dynamic multipathing (for active/active storage devices), or numerous other features that require the additional licenses. CVM is an enhanced version of the VxVM volume manager that is specifically designed for use in ServiceGuard OPS Edition clusters. When installed with the VERITAS Volume Manager for HP-UX (B9116AA), the CVM add-on product provides most of the enhanced VxVM features in a clustered environment. CVM is truly cluster-aware, obtaining information about cluster membership from ServiceGuard OPS Edition directly. Cluster information is provided via a special system multi-node package known as VxVM-CVM-pkg, which runs on all nodes in the cluster. The cluster must be up and must be running this package in order to configure VxVM disk groups for use with CVM.

CVM allows you to activate storage on one node at a time, or you can perform write activation on one node and read activation on another node at the same time (for example, allowing backups). CVM provides full mirroring and dynamic multipathing (DMP) for clusters.

CVM can be used in clusters that:

run Oracle Parallel Server with advanced storage management features or with more than two nodes. For ServiceGuard OPS Edition customers, a CVM two-node shared write activation license is included along with the Base VxVM Volume Manager. However, mirroring and four-node capability require additional licenses.
will run applications that require fast disk group activation after package failover.
require activation on more than one node at a time, for example to perform a backup from one node while a package using the volume is active on another node. In this case, the package using the disk group would have the disk group active in exclusive write mode while the node that is doing the backup would have the disk group active in shared read mode.

CVM is supported on 4 nodes or fewer at this release. Shared storage devices must be connected to all nodes in the cluster, whether or not the node accesses data on the device.




	NOTE: In order to activate a disk group on more than two nodes (for example with a 3 or 4 node OPS cluster), the CVM full license is required. The installation of the CVM full license also requires the VxVM full license to be installed. There is an additional cost for each license.

Cluster Startup Time with CVM

With CVM, all shared disk groups (DGs) are imported when the VxVM-CVM-pkg control script completes. Depending on the number of DGs, the number of nodes and the configuration of these (number of disks, volumes, etc.) this can take some time (current timeout value for this package is 3 minutes but for larger configurations this may have to be increased). Any failover package that uses a CVM DG will not start until the VxVM-CVM-pkg is up. Note that this delay does not affect package failover time; it is a one-time overhead cost at cluster startup.

Propagation of Disk Groups with CVM

With CVM, disk groups are created on one cluster node known as the CVM master node. The cluster has to be running to create shared disk groups. CVM will validate that each node can see each disk and will not allow invalid DGs to be created.

Single Heartbeat Subnet Required with CVM

It is normally recommended that you configure all subnets that interconnect cluster nodes as heartbeat networks, since this increases protection against multiple faults at no additonal cost. However, if you will be using the VERITAS Cluster Volume Manager (CVM), only a single heartbeat subnet is allowed. When the VXVM-CVM-pkg is added (as shown in Chapter 5), cmcheckconf and cmapplyconf will check to ensure that only a single heartbeat subnet is configured. If more heartbeat subnets are configured, the cmcheckconf and cmapplyconf will fail.

This restriction is due to the design of the VERITAS Cluster Daemon (vxclustd), which is started within the VxVM-CVM-pkg. This daemon is only able to communicate information to cluster nodes on a single IP connection on a single subnet. Moreover, communication between nodes in the cluster must be the same for the Veritas Cluster Daemon as for the ServiceGuard cluster daemon cmcld. Therefore, cmcld must also only have the use of a single heartbeat subnet.

There are tradeoffs in only using a single subnet. If the heartbeat is not heard for more than a NODE_TIMEOUT interval, the cluster will re-form. Depending on the length of the disruption, this could cause one or all of the cluster nodes to TOC. This is most commonly seen when there is heavy traffic on the subnet preventing the heartbeat packets from being transmitted. This can also been seen when there are network configuration problems or network hardware problems that are not easily recovered and may not be detected by ServiceGuard or APA. The most common way to avoid problems in these scenarios is to utilize multiple heartbeat networks.

In the case of CVM, the single heartbeat subnet should be configured with standby LANS or as a group of aggregated ports to help minimize the possibility of system or cluster failures.

Comparison of Volume Managers

The following table summarizes some of the advantages and disadvantages of the volume managers that are currently available.

Table 3-5 Volume Managers Supported with ServiceGuard OPS Edition

Product	Pros	Cons
Logical Volume Manager (LVM)	Legacy system is robust and familiar to HP-UX users Existing packages do not need to be changed Supports up to 16 nodes per cluster in exclusive mode Supports multiple heartbeat subnets	Lacks flexibility and extended features of other volume managers
Shared Logical Volume Manager (SLVM)	Provided free with ServiceGuard OPS Edition for multi-node activation of disks with OPS data Supports up to eight nodes in shared read/write mode per cluster (more may be supported later via an SLVM patch). Supports exclusive activation Supports multiple heartbeat subnets	Lacks flexibility and extended features of other volume managers Limited mirroring support All shared volume groups in the cluster must be deactivated before online node configuration.
Base-VXVM	Software is free with HP-UX 11i and later Supports multiple heartbeat subnets	Limited feature set
VERITAS Volume Manager—B9116AA (VxVM)	Disk group configuration from any node Enhanced set of volume management features, including software mirroring, RAID 0/1, RAID 5, and DMP for active/active storage devices Cluster startup time is faster than with CVM. Supports up to 16 nodes in exclusive write mode per cluster Supports multiple heartbeat subnets	Does not support activation on multiple nodes Package startup may be slower than with CVM Enhanced features require purchase of additional license
VERITAS Cluster Volume Manager—B9117AA (CVM)	Enhanced set of volume management features for OPS and non-OPS data, including software mirroring and RAID 0/1 Package startup time is faster than with VxVM Supports exclusive activation Supports shared read/write activation on up to 4 nodes at the same time for OPS instances Shared disk groups can remain active during online node configurations	Disk groups must be configured on a master node Can only be used with up to 4 cluster nodes Cluster startup may be slower than with VxVM Requires purchase of additional license No support for striped mirrors or RAID 5 Supports only a single heartbeat subnets