Creating the Storage Infrastructure and Filesystems with VERITAS Cluster Volume Manager (CVM)

In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This can be done with Logical Volume Manager (LVM), VERITAS Volume Manager (VxVM), or VERITAS Cluster Volume Manager (CVM) before cluster configuration. It can also be done with CVM configuration (with or without Cluster File System) after cluster configuration. You can also use a mixture of volume types, depending on your needs.

Appendix G “Migrating from LVM to VxVM Data Storage ” for a discussion of migration.

This section has information about configuring the VERITAS Cluster Volume Manager 3.5 and 4.1 without VERITAS CFS (Cluster File System). The configuration may be needed to set up raw devices for Serviceguard Extension for RAC.

The previous section (“Creating a Storage Infrastructure with VERITAS Cluster File System (CFS)”) has information about configuring a cluster with CFS and with CVM 4.1.

Both solutions - with and without CFS - use many of the same commands, but the processes are in a slightly different order.

Before starting, make sure the directory in which VxVM commands are stored (/usr/lib/vxvm/bin) is in your path. Once you have created the root disk group with vxinstall, you can use VxVM commands or the VERITAS Storage Administrator GUI, VEA, to carry out configuration tasks. Detailed instructions for running vxinstall are given in the VERITAS Volume Manager Release Notes. For more information, refer to the VERITAS Volume Manager Administrator’s Guide.

Separate procedures are given below for:

Initializing the Volume Manager
Preparing the Cluster for Use with CVM
Creating Disk Groups for Shared Storage
Creating File Systems with CVM

For more information, including details about configuration of plexes (mirrors), multipathing, and RAID, refer to the HP-UX documentation for the VERITAS Volume Manager. See the documents for HP Serviceguard Storage Management Suite posted at http://docs.hp.com.

Initializing the VERITAS Volume Manager

If you are about to create disk groups for the first time, you need to initialize the Volume Manager.

Use the following command after installing VxVM/CVM on each node:

# vxinstall

This displays a menu-driven program that steps you through the VxVM/CVM initialization sequence.

In CVM 3.5, you must create a disk group known as rootdg that contains at least one disk. From the main menu, choose the “Custom” option, and specify the disk you wish to include in rootdg.




	IMPORTANT: The rootdg in version 3.5 of VERITAS Volume Manager is not the same as the HP-UX root disk if an LVM volume group is used for the HP-UX root filesystem (/). Note also that rootdg cannot be used for shared storage. However, rootdg can be used for other local filesystems (e.g., `/export/home`), so it need not be wasted. Note that you should create a root disk group only once on each node.

CVM 4.1 does not require that you create the special VERITAS rootdg disk.

Preparing the Cluster for Use with CVM

In order to use the VERITAS Cluster Volume Manager (CVM), you need a cluster that is running with a Serviceguard-supplied CVM system multi-node package. This means that the cluster must already be configured and running before you create disk groups.

Configure system multi-node and multi-node packages with the command line, not through Serviceguard Manager. In Serviceguard Manager, system multi-node and multi-node packages are not displayed on the map and tree, like failover packages. These clusterwide packages’ properties have a special tab in the cluster properties, and their admin menu is available when you select their cluster.




	NOTE: Cluster configuration is described in the previous section, “Configuring the Cluster ”.

Check the heartbeat configuration. The CVM 3.5 heartbeat requirement is different from version 4.1:

With CVM 3.5, prepare the cluster for CVM disk group configuration you can configure only one heartbeat subnet in the cluster.
With CVM 4.1, the cluster can have multiple heartbeats.

Neither version can use Auto Port Aggregation, Infiniband, or VLAN interfaces as a heartbeat subnet.

The VERITAS cluster volumes are managed by a Serviceguard-supplied system multi-node package which runs on all nodes at once, and cannot failover. In CVM 3.5, Serviceguard creates the VxVM-CVM-pkg. In CVM 4.1, Serviceguard creates the SG-CFS-pkg.

The SG-CFS-pkg package has the following responsibilities:

Maintain VERITAS configuration files /etc/llttab, /etc/llthosts, /etc/gabtab
Launch required services: cmvxd, cmvxpingd, vxfsckd
Start/halt VERITAS process in the proper order: llt, gab, vxfen, odm, cvm, cfs

The following commands create the system multi-node package that communicates cluster information to CVM:

VERITAS CVM 3.5:
# cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf
VERITAS CVM 4.1: If you are not using VERITAS Cluster File System, use the cmapplyconf command. (If you are using CFS, you will set up CVM as part of the CFS components.):
# cmapplyconf -P /etc/cmcluster/cfs/SG-CFS-pkg.conf
Begin package verification ... Modify the package configuration ([y]/n)? Y Completed the cluster update

You can confirm this using the cmviewcl command. This output fshows results of the CVM 3.5 command above.

CLUSTER      STATUS       
example      up           
 
NODE         STATUS       STATE 
ftsys9       up           running 
ftsys10      up           running 
 
MULTI_NODE_PACKAGES:
 
PACKAGE      STATUS     STATE    AUTO_RUN   SYSTEM 
VxVM-CVM-pkg  up        running  enabled     yes




	NOTE: Do not edit system multi-node package configuration files, such as `VxVM-CVM-pkg.conf` and `SG-CFS-pkg.conf`. Create and modify configuration using the cfs admin commands listed in Appendix A.

Starting the Cluster and Identifying the Master Node

If it is not already running, start the cluster. This will automatically activate the special CVM package:

# cmruncl

When CVM starts up, it selects a master node, and this is the node from which you must issue the disk group configuration commands. To determine the master node, issue the following command from each node in the cluster:

# vxdctl -c mode

One node will identify itself as the master. Create disk groups from this node.

Initializing Disks for CVM

You need to initialize the physical disks that will be employed in CVM disk groups. If a physical disk has been previously used with LVM, you should use the pvremove command to delete the LVM header data from all the disks in the volume group (this is not necessary if you have not previously used the disk with LVM).

To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example:

# /usr/lib/vxvm/bin/vxdisksetup -i c4t3d4

Creating Disk Groups

Use the following steps to create disk groups.

Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the following example:
# vxdg -s init logdata c0t3d2

Verify the configuration with the following command:

# vxdg list

NAME         STATE                  ID
 
rootdg        enabled             971995699.1025.node1
logdata       enabled,shared      972078742.1084.node2

Activate the disk group, as follows, before creating volumes:
# vxdg -g logdata set activation=ew

Creating Volumes

Use the vxassist command to create volumes, as in the following example:

# vxassist -g logdata make log_files 1024m

This command creates a 1024 MB volume named log_files in a disk group named logdata. The volume can be referenced with the block device file /dev/vx/dsk/logdata/log_files or the raw (character) device file /dev/vx/rdsk/logdata/log_files.

Verify the configuration with the following command:

# vxdg list

Mirror Detachment Policies with CVM

The default CVM disk mirror detachment policy is global, which means that as soon as one node cannot see a specific mirror copy (plex), all nodes cannot see it as well. The alternate policy is local, which means that if one node cannot see a specific mirror copy, then CVM will deactivate access to the volume for that node only.

The global policy is recommended, because it can ensure all nodes are accessing the same current data. If you use local, it can cause problems if one node cannot update one of the mirror copies and the data on that copy goes stale. If any of the other nodes tries to read from that mirror copy, they would read the stale data. This can be avoided with the global option, because all nodes will only use the current mirror copy, so they will all read consistent data.

This policy can be re-set on a disk group basis by using the vxedit command, as follows:

# vxedit set diskdetpolicy=[global|local] <DiskGroupName>




	NOTE: The specific commands for creating mirrored and multi-path storage using CVM are described in the HP-UX documentation for the VERITAS Volume Manager, posted at http://docs.hp.com.

Creating File Systems

If your installation uses filesystems, create them next. Use the following commands to create a filesystem for mounting on the logical volume just created:

Create the filesystem on the newly created volume:
# newfs -F vxfs /dev/vx/rdsk/logdata/log_files
Create a directory to mount the volume:
# mkdir /logs
Mount the volume:
# mount /dev/vx/dsk/logdata/log_files /logs
Check to make sure the filesystem is present, then unmount it:
# umount /logs
Use the following command to deactivate the disk group:
# vxdg -g logdata set activation=off
After creating units of storage with VxVM commands, you need to specify the CVM disk groups in each package configuration file. If you configure your package with Serviceguard Manager, you can select from a list of discovered disk groups. If you use the ASCII configuration file, use one STORAGE_GROUP parameter for each disk group the package will use.
If you configure your package in Serviceguard Manager, it can create your control scripts in guided mode. If you want to edit the control script yourself, identify the CVM disk groups, filesystems, logical volumes, and mount points in the ASCII package control script file.
The package configuration process is described in “Creating the Package Configuration ” and “Creating the Package Control Script”.
NOTE: Unlike LVM volume groups, CVM disk groups are not entered in the cluster configuration file, they are entered in the package configuration file only.

Adding Disk Groups to the Package Configuration

After creating units of storage with VxVM commands, you need to specify the CVM disk groups in each package configuration ASCII file. Use one DISK_GROUP parameter for each disk group the package will use. You also need to identify the CVM disk groups, filesystems, logical volumes, and mount options in the package control script. The package configuration process is described in detail in Chapter 6.