Creating the Storage Infrastructure and Filesystems with LVM and VxVM

In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done several ways:

for Logical Volume Manager, see “Creating a Storage Infrastructure with LVM”
for VERITAS Volume Manager “Creating a Storage Infrastructure with VxVM”
for VERITAS Cluster File System with CVM, “Creating a Storage Infrastructure with VERITAS Cluster File System (CFS)”
for VERITAS Cluster Volume Manager, see “Creating the Storage Infrastructure and Filesystems with VERITAS Cluster Volume Manager (CVM)”

You can also use a mixture of volume types, depending on your needs.

LVM and VxVM configuration is done before configuring the cluster. CVM and CFS configuration is done after configuring the cluster.

Creating a Storage Infrastructure with LVM

This section describes storage configuration with LVM. Separate procedures are given for the following:

Creating Volume Groups for Mirrored Individual Disks
Creating Volume Groups for Disk Arrays Using PV Links
Distributing Volume Groups to Other Nodes

The Event Monitoring Service HA Disk Monitor provides the capability to monitor the health of LVM disks. If you intend to use this monitor for your mirrored disks, you should configure them in physical volume groups. For more information, refer to the manual Using High Availability Monitors.

Creating Volume Groups for Mirrored Individual Data Disks

The procedure described in this section uses physical volume groups for mirroring of individual disks to ensure that each logical volume is mirrored to a disk on a different I/O bus. This kind of arrangement is known as PVG-strict mirroring. It is assumed that your disk hardware is already configured in such a way that a disk to be used as a mirror copy is connected to each node on a different bus than the bus that is used for the other (primary) copy.

For more information on using LVM, refer to the HP-UX Managing Systems and Workgroups manual.

Using SAM to Create Volume Groups and Logical Volumes

You can use SAM to prepare the volume group and logical volume structure needed for HA packages. In SAM, choose the “Disks and File Systems Area.” Then use the following procedure for each volume group and filesystem you are using with the package:

Select the Volume Groups subarea.
From the Actions menu, choose Create or Extend.
Choose the first physical disk you wish to use in the volume group by selecting it from the list of available disks.
Enter the volume group name, e.g., vgdatabase.
Choose Create or Extend Volume Group.
Choose Add New Logical Volumes.
When adding logical volumes to the volume group, ensure that you are creating mirrored logical volumes with PVG strict allocation.
Specify the filesystem that is to be mounted on the volume group, for example, /mnt1.
Repeat the procedure for additional volume groups, logical volumes, and filesystems.

Skip ahead to the section “Deactivating the Volume Group”.

Using LVM Commands to Create Volume Groups and Logical Volumes

If your volume groups have not been set up, use the procedure in the next sections. If you have already done LVM configuration, skip ahead to the section “Configuring the Cluster.”

Selecting Disks for the Volume Group

Obtain a list of the disks on both nodes and identify which device files are used for the same disk on both. Use the following command on each node to list available disks as they are known to each system:

# lssf /dev/dsk/*

In the following examples, we use /dev/rdsk/c1t2d0 and /dev/rdsk/c0t2d0, which happen to be the device names for the same disks on both ftsys9 and ftsys10. In the event that the device file names are different on the different nodes, make a careful note of the correspondences.

Creating Physical Volumes

On the configuration node (ftsys9), use the pvcreate command to define disks as physical volumes. This only needs to be done on the configuration node. Use the following commands to create two physical volumes for the sample configuration:

# pvcreate -f /dev/rdsk/c1t2d0 
# pvcreate -f /dev/rdsk/c0t2d0

Creating a Volume Group with PVG-Strict Mirroring

Use the following steps to build a volume group on the configuration node (ftsys9). Later, the same volume group will be created on other nodes.

First, set up the group directory for vgdatabase:
# mkdir /dev/vgdatabase
Next, create a control file named group in the directory /dev/vgdatabase, as follows:
# mknod /dev/vgdatabase/group c 64 0xhh0000
The major number is always 64, and the hexadecimal minor number has the form
0xhh0000
where hh must be unique to the volume group you are creating. Use a unique minor number that is available across all the nodes for the mknod command above. (This will avoid further reconfiguration later, when NFS-mounted logical volumes are created in the VG.)
Use the following command to display a list of existing volume groups:
# ls -l /dev/*/group
Create the volume group and add physical volumes to it with the following commands:
# vgcreate -g bus0 /dev/vgdatabase /dev/dsk/c1t2d0 # vgextend -g bus1 /dev/vgdatabase /dev/dsk/c0t2d0
The first command creates the volume group and adds a physical volume to it in a physical volume group called bus0. The second command adds the second drive to the volume group, locating it in a different physical volume group named bus1. The use of physical volume groups allows the use of PVG-strict mirroring of disks and PV links.
Repeat this procedure for additional volume groups.

Creating Logical Volumes

Use the following command to create logical volumes (the example is for /dev/vgdatabase):

# lvcreate -L 120 -m 1 -s g /dev/vgdatabase

This command creates a 120 MB mirrored volume named lvol1. The name is supplied by default, since no name is specified in the command. The -s g option means that mirroring is PVG-strict, that is, the mirror copies of data will be in different physical volume groups.




	NOTE: If you are using disk arrays in RAID 1 or RAID 5 mode, omit the `-m 1` and `-s g` options.

Creating File Systems

If your installation uses filesystems, create them next. Use the following commands to create a filesystem for mounting on the logical volume just created:

Create the filesystem on the newly created logical volume:
# newfs -F vxfs /dev/vgdatabase/rlvol1
Note the use of the raw device file for the logical volume.
Create a directory to mount the disk:
# mkdir /mnt1
Mount the disk to verify your work:
# mount /dev/vgdatabase/lvol1 /mnt1
Note the mount command uses the block device file for the logical volume.
Verify the configuration:
# vgdisplay -v /dev/vgdatabase

Creating Volume Groups for Disk Arrays Using PV Links

If you are configuring volume groups that use mass storage on HP's HA disk arrays, you should use redundant I/O channels from each node, connecting them to separate ports on the array. Then you can define alternate links (also called PV links) to the LUNs or logical disks you have defined on the array. In SAM, choose the type of disk array you wish to configure, and follow the menus to define alternate links.

The following example shows how to configure alternate links using LVM commands. In the example, the following disk configuration is assumed:

8/0.15.0 /dev/dsk/c0t15d0  /* I/O Channel 0 (8/0) SCSI address 15 LUN 0 */
8/0.15.1 /dev/dsk/c0t15d1  /* I/O Channel 0 (8/0) SCSI address 15 LUN 1 */
8/0.15.2 /dev/dsk/c0t15d2  /* I/O Channel 0 (8/0) SCSI address 15 LUN 2 */
8/0.15.3 /dev/dsk/c0t15d3  /* I/O Channel 0 (8/0) SCSI address 15 LUN 3 */
8/0.15.4 /dev/dsk/c0t15d4  /* I/O Channel 0 (8/0) SCSI address 15 LUN 4 */
8/0.15.5 /dev/dsk/c0t15d5  /* I/O Channel 0 (8/0) SCSI address 15 LUN 5 */
 
10/0.3.0 /dev/dsk/c1t3d0   /* I/O Channel 1 (10/0) SCSI address 3 LUN 0 */
10/0.3.1 /dev/dsk/c1t3d1   /* I/O Channel 1 (10/0) SCSI address 3 LUN 1 */
10/0.3.2 /dev/dsk/c1t3d2   /* I/O Channel 1 (10/0) SCSI address 3 LUN 2 */
10/0.3.3 /dev/dsk/c1t3d3   /* I/O Channel 1 (10/0) SCSI address 3 LUN 3 */
10/0.3.4 /dev/dsk/c1t3d4   /* I/O Channel 1 (10/0) SCSI address 3 LUN 4 */
10/0.3.5 /dev/dsk/c1t3d5   /* I/O Channel 1 (10/0) SCSI address 3 LUN 5 */

Assume that the disk array has been configured, and that both the following device files appear for the same LUN (logical disk) when you run the ioscan command:

/dev/dsk/c0t15d0
/dev/dsk/c1t3d0

Use the following steps to configure a volume group for this logical disk:

First, set up the group directory for vgdatabase:
# mkdir /dev/vgdatabase
Next, create a control file named group in the directory /dev/vgdatabase, as follows:
# mknod /dev/vgdatabase/group c 64 0xhh0000
The major number is always 64, and the hexadecimal minor number has the form
0xhh0000
where hh must be unique to the volume group you are creating. Use a unique number that is available across all the nodes. (This will avoid further reconfiguration later, when NFS-mounted logical volumes will be created in the VG.)
Use the following command to display a list of existing group files:
# ls -l /dev/*/group
Use the pvcreate command on one of the device files associated with the LUN to define the LUN to LVM as a physical volume.
# pvcreate /dev/rdsk/c0t15d0
It is only necessary to do this with one of the device file names for the LUN.

Use the following commands to create the volume group itself:

# vgcreate /dev/vgdatabase /dev/dsk/c0t15d0 
# vgextend /dev/vgdatabase /dev/dsk/c1t3d0

You can now use the vgdisplay -v command to see the primary and alternate links. LVM will now recognize the I/O channel represented by /dev/dsk/c0t15d0 as the primary link to the disk; if the primary link fails, LVM will automatically switch to the alternate I/O channel represented by /dev/dsk/c1t3d0.

To create logical volumes, use the procedure described in the previous section, “Creating Logical Volumes.”

Distributing Volume Groups to Other Nodes

After creating volume groups for cluster data, you must make them available to any cluster node that will need to activate the volume group. The cluster lock volume group must be made available to all nodes.

Deactivating the Volume Group

At the time you create the volume group, it is active on the configuration node (ftsys9, for example). Before setting up the volume group for use on other nodes, you must first unmount any filesystems that reside on the volume group, then deactivate it. At run time, volume group activation and filesystem mounting are done through the package control script.

Continuing with the example presented in earlier sections, do the following on ftsys9:

# umount /mnt1 
# vgchange -a n /dev/vgdatabase

Distributing the Volume Group with LVM Commands

Use the following commands to set up the same volume group on another cluster node. In this example, the commands set up a new volume group on ftsys10 which will hold the same physical volume that was available on ftsys9. You must carry out the same procedure separately for each node on which the volume group's package can run.

To set up the volume group on ftsys10, use the following steps:

On ftsys9, copy the mapping of the volume group to a specified file.
# vgexport -p -s -m /tmp/vgdatabase.map /dev/vgdatabase

Still on ftsys9, copy the map file to ftsys10:

# rcp /tmp/vgdatabase.map ftsys10:/tmp/vgdatabase.map

On ftsys10, create the volume group directory:
# mkdir /dev/vgdatabase
Still on ftsys10, create a control file named group in the directory /dev/vgdatabase, as follows:
# mknod /dev/vgdatabase/group c 64 0xhh0000
Use the same minor number as on ftsys9. Use the following command to display a list of existing volume groups:
# ls -l /dev/*/group
Import the volume group data using the map file from node ftsys9. On node fsys10, enter:
# vgimport -s -m /tmp/vgdatabase.map /dev/vgdatabase
Note that the disk device names on ftsys10 may be different from their names on ftsys9. You should check to ensure that the physical volume names are correct throughout the cluster.
When the VG can be activated on this node, perform a vgcfgbackup in the unlikely event that a vgcfgrestore must be performed on this node because of a disaster on the primary node and an LVM problem with the volume group. Do this as shown in the example below:
# vgchange -a y /dev/vgdatabase # vgcfgbackup /dev/vgdatabase # vgchange -a n /dev/vgdatabase

If you are using mirrored individual disks in physical volume groups, check the /etc/lvmpvg file to ensure that each physical volume group contains the correct physical volume names for ftsys10.




	NOTE: When you use PVG-strict mirroring, the physical volume group configuration is recorded in the `/etc/lvmpvg` file on the configuration node. This file defines the physical volume groups which are the basis of mirroring and indicate which physical volumes belong to each PVG. Note that on each cluster node, the `/etc/lvmpvg` file must contain the correct physical volume names for the PVG’s disks as they are known on that node. Physical volume names for the same disks may not be the same on different nodes. After distributing volume groups to other nodes, you must ensure that each node’s `/etc/lvmpvg` file correctly reflects the contents of all physical volume groups on that node. Refer to the following section, “Making Physical Volume Group Files Consistent.”

Make sure that you have deactivated the volume group on ftsys9. Then enable the volume group on ftsys10:
# vgchange -a y /dev/vgdatabase
Create a directory to mount the disk:
# mkdir /mnt1
Mount and verify the volume group on ftsys10:
# mount /dev/vgdatabase/lvol1 /mnt1
Unmount the volume group on ftsys10:
# umount /mnt1
Deactivate the volume group on ftsys10:
# vgchange -a n /dev/vgdatabase

Making Physical Volume Group Files Consistent

Skip ahead to the next section if you do not use physical volume groups for mirrored individual disks in your disk configuration.

Different volume groups may be activated by different subsets of nodes within a Serviceguard cluster. In addition, the physical volume name for any given disk may be different on one node than it is on another. For these reasons, you must carefully merge the /etc/lvmpvg files on all nodes so that each node has a complete and consistent view of all cluster-aware disks as well as of its own private (non-cluster-aware) disks. To make merging the files easier, be sure to keep a careful record of the physical volume group names on the volume group planning worksheet (described in the “Planning” chapter).

Use the following procedure to merge files between the configuration node (ftsys9) and a new node (ftsys10) to which you are importing volume groups:

Copy /etc/lvmpvg from ftsys9 to /etc/lvmpvg.new on ftsys10.
If there are volume groups in /etc/lvmpvg.new that do not exist on ftsys10, remove all entries for that volume group from /etc/lvmpvg.new.
If /etc/lvmpvg on ftsys10 contains entries for volume groups that do not appear in /etc/lvmpvg.new, then copy all PVG entries for that volume group to /etc/lvmpvg.new.
Adjust any physical volume names in /etc/lvmpvg.new to reflect their correct names on ftsys10.
On ftsys10, copy /etc/lvmpvg to /etc/lvmpvg.old to create a backup. Copy /etc/lvmvpg.new to /etc/lvmpvg on ftsys10.

Creating Additional Volume Groups

The foregoing sections show in general how to create volume groups and logical volumes for use with Serviceguard. Repeat the procedure for as many volume groups as you need to create, substituting other volume group names, logical volume names, and physical volume names. Pay close attention to the disk device names. For example, /dev/dsk/c0t2d0 on one node may not be /dev/dsk/c0t2d0 on another node.

Creating a Storage Infrastructure with VxVM

In addition to configuring the cluster, you create the appropriate logical volume infrastructure to provide access to data from different nodes. This is done with Logical Volume Manager (LVM), VERITAS Volume Manager (VxVM), or VERITAS Cluster Volume Manager (CVM). You can also use a mixture of volume types, depending on your needs. LVM and VxVM configuration are done before cluster configuration, and CVM configuration is done after cluster configuration.

For a discussion of migration from LVM to VxVM storage, refer to Appendix G.

This section shows how to configure new storage using the command set of the VERITAS Volume Manager (VxVM). Once you have created the root disk group (described next), you can use VxVM commands or the Storage Administrator GUI, VEA, to carry out configuration tasks. Details are given in the VERITAS Volume Manager for HP-UX Release Notes. For more information, refer to the VERITAS Enterprise Administrator (VEA 500 Series) Getting Started. If you are using commands, refer to the VxVM man pages.

Initializing the VERITAS Cluster Volume Manager 3.5

With CVM 3.5, if you are about to create disk groups for the first time, you need to initialize the Volume Manager. This is done by creating a disk group known as rootdg that contains at least one disk. Use the following command once only, immediately after installing VxVM on each node:

# vxinstall

This displays a menu-driven program that steps you through the VxVM initialization sequence. From the main menu, choose the “Custom” option, and specify the disk you wish to include in rootdg.




	IMPORTANT: The rootdg for the VERITAS Cluster Volume Manager 3.5 is not the same as the HP-UX root disk if an LVM volume group is used for the HP-UX root disk filesystem. Note also that rootdg cannot be used for shared storage. However, rootdg can be used for other local filesystems (e.g., `/export/home`), so it need not be wasted. (CVM 4.1 does not have this restriction.) Note that you should create a rootdg disk group only once on each node.

Converting Disks from LVM to VxVM

You can use the vxvmconvert(1m) utility to convert LVM volume groups into VxVM disk groups. Before you can do this, the volume group must be deactivated, which means that any package that uses the volume group must be halted. Follow the conversion procedures outlined in the VERITAS Volume Manager Migration Guide. Before you start, be sure to create a backup of each volume group’s configuration with the vgcfgbackup command, and make a backup of the data in the volume group. Appendix G “Migrating from LVM to VxVM Data Storage ” for additional details about conversion.

Initializing Disks for VxVM

You need to initialize the physical disks that will be employed in VxVM disk groups. To initialize a disk, log on to one node in the cluster, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example:

# /usr/lib/vxvm/bin/vxdisksetup -i c0t3d2

Initializing Disks Previously Used by LVM

If a physical disk has been previously used with LVM, you should use the pvremove command to delete the LVM header data from all the disks in the volume group. In addition, if the LVM disk was previously used in a cluster, you have to re-initialize the disk with the pvcreate -f command to remove the cluster ID from the disk.




	NOTE: These commands make the disk and its data unusable by LVM, and allow it to be initialized by VxVM. (The commands should only be used if you have previously used the disk with LVM and do not want to save the data on it.)

You can remove LVM header data from the disk as in the following example (note that all data on the disk will be erased):# pvremove /dev/rdsk/c0t3d2

Then, use the vxdiskadm program to initialize multiple disks for VxVM, or use the vxdisksetup command to initialize one disk at a time, as in the following example:

# /usr/lib/vxvm/bin/vxdisksetup -i c0t3d2

Creating Disk Groups

Use vxdiskadm, or use the vxdg command, to create disk groups, as in the following example:

# vxdg init logdata c0t3d2

Verify the configuration with the following command:

# vxdg list

NAME         STATE                  ID
 
rootdg        enabled             971995699.1025.node1
logdata       enabled             972078742.1084.node1

Creating Volumes

Use the vxassist command to create logical volumes. The following is an example:

# vxassist -g logdata make log_files 1024m

This command creates a 1024 MB volume named log_files in a disk group named logdata. The volume can be referenced with the block device file /dev/vx/dsk/logdata/log_files or the raw (character) device file /dev/vx/rdsk/logdata/log_files. Verify the configuration with the following command:

# vxprint -g logdata

The output of this command is shown in the following example:

TY  NAME       ASSOC        KSTATE   LENGTH   PLOFFS   STATE   TUTILO  PUTILO
 
v   logdata    fsgen        ENABLED  1024000          ACTIVE
pl  logdata-01 system       ENABLED  1024000          ACTIVE




	NOTE: The specific commands for creating mirrored and multi-path storage using VxVM are described in the VERITAS Volume Manager Reference Guide.

Creating File Systems

If your installation uses filesystems, create them next. Use the following commands to create a filesystem for mounting on the logical volume just created:

Create the filesystem on the newly created volume:
# newfs -F vxfs /dev/vx/rdsk/logdata/log_files
Create a directory to mount the volume:
# mkdir /logs
Mount the volume:
# mount /dev/vx/dsk/logdata/log_files /logs
Check to make sure the filesystem is present, then unmount the filesystem:
# umount /logs

Deporting Disk Groups

After creating the disk groups that are to be used by Serviceguard packages, use the following command with each disk group to allow the disk group to be deported by the package control script on other cluster nodes:

# vxdg deport <DiskGroupName>

where <DiskGroupName> is the name of the disk group that will be activated by the control script.

When all disk groups have been deported, you must issue the following command on all cluster nodes to allow them to access the disk groups:

# vxdctl enable

Re-Importing Disk Groups

After deporting disk groups, they are not available for use on the node until they are imported again either by a package control script or with a vxdg import command. If you need to import a disk group manually for maintenance or other purposes, you import it, start up all its logical volumes, and mount filesystems as in the following example:

# vxdg import dg_01

# vxvol -g dg_01 startall

# mount /dev/vx/dsk/dg_01/myvol /mountpoint




	NOTE: Unlike LVM volume groups, VxVM disk groups are not entered in the cluster ASCII configuration file, and they are not entered in the package ASCII configuration file.

Clearimport at System Reboot Time

At system reboot time, the cmcluster RC script does a vxdisk clearimport on all disks formerly imported by the system, provided they have the noautoimport flag set, and provided they are not currently imported by another running node. The clearimport clears the host ID on the disk group, to allow any node that is connected to the disk group to import it when the package moves from one node to another.

Using the clearimport at reboot time allows Serviceguard to clean up following a node failure, for example, a system crash during a power failure. Disks that were imported at the time of the failure still have the node’s ID written on them, and this ID must be cleared before the rebooting node or any other node can import them with a package control script.

Note that the clearimport is done for disks previously imported with noautoimport set on any system that has Serviceguard installed, whether it is configured in a cluster or not.

Creating the Storage Infrastructure and Filesystems with LVM and VxVM

Technical documentation

» Table of Contents

» Index

Creating a Storage Infrastructure with LVM

Creating Volume Groups for Mirrored Individual Data Disks

Using SAM to Create Volume Groups and Logical Volumes

Using LVM Commands to Create Volume Groups and Logical Volumes

Selecting Disks for the Volume Group

Creating Physical Volumes

Creating a Volume Group with PVG-Strict Mirroring

Creating Logical Volumes

Creating File Systems

Creating Volume Groups for Disk Arrays Using PV Links

Distributing Volume Groups to Other Nodes

Deactivating the Volume Group

Distributing the Volume Group with LVM Commands

Making Physical Volume Group Files Consistent

Creating Additional Volume Groups

Creating a Storage Infrastructure with VxVM

Initializing the VERITAS Cluster Volume Manager 3.5

Converting Disks from LVM to VxVM

Initializing Disks for VxVM

Initializing Disks Previously Used by LVM

Creating Disk Groups

Creating Volumes

Creating File Systems

Deporting Disk Groups

Re-Importing Disk Groups

Clearimport at System Reboot Time