Creating an OPS Storage Infrastructure with LVM

Creating Mirrors of Root Logical Volumes

It is highly recommended that you use mirrored root volumes on all cluster nodes. If you are using a disk array for your root logical volumes, skip ahead to the section, "“Creating OPS Volume Groups on Disk Arrays ”." If you are using a standalone root disk, it should be mirrored as shown here.

The following procedure assumes that you are using separate boot and root volumes; you create a mirror of the boot volume (/dev/vg00/lvol1), root volume (/dev/vg00/lvol3), and primary swap (/dev/vg00/lvol2). The procedure cannot be carried out with SAM. In this example and in the following commands, /dev/dsk/c4t5d0 is the primary disk and /dev/dsk/c4t6d0 is the mirror; be sure to use the correct device file names for the root disks on your system.

Creating Cluster Lock Disks

The cluster lock disk is configured on any volume group that is physically connected to all cluster nodes. This volume group may contain data that is used by packages.

When you are using dual cluster lock disks, it is required that the default IO timeout values are used for the cluster lock physical volumes. Changing the IO timeout values for the cluster lock physical volumes can prevent the nodes in the cluster from detecting a failed lock disk within the alloted time period which can prevent cluster re-formations from succeeding. For existing IO timeout value, run the following command:

# pvdisplay <lock device file name>

The IO Timeout value should be displayed as "default." To set the IO Timeout back to the default value, run the command:

# pvchange -t 0 <lock device file name>

The use of a dual cluster lock is only allowed with specific hardware configurations. See "Dual Cluster Lock" in Chapter 3.

Backing Up Cluster Lock Disk Information

After you configure the cluster and create the cluster lock volume group and physical volume, you should create a backup of the volume group configuration data on each lock volume group. Use the vgcfgbackup command for each lock volume group you have configured, and save the backup file in case the lock configuration must be restored to a new disk with the vgcfgrestore command following a disk failure.

	NOTE: You must use the vgcfgbackup and vgcfgrestore commands to back up and restore the lock volume group configuration data regardless of whether you use SAM or HP-UX commands to create the lock volume group.

Building Volume Groups for OPS on Mirrored Disks

The procedure described in this section uses physical volume groups for mirroring of individual disks to ensure that each logical volume is mirrored to a disk on a different I/O bus. This kind of arrangement is known as PVG-strict mirroring. It is assumed that your disk hardware is already configured in such a way that a disk to be used as a mirror copy is connected to each node on a different bus than the bus that is used for the other (primary) copy.

For more information on using LVM, refer to the HP-UX Managing Systems and Workgroups manual.

Creating Volume Groups and Logical Volumes

If your volume groups have not been set up, use the procedure in the next sections. If you have already done LVM configuration, skip ahead to the section "Configuring the Cluster."

Selecting Disks for the Volume Group

Obtain a list of the disks on both nodes and identify which device files are used for the same disk on both. Use the following command on each node to list available disks as they are known to each system:

# lssf /dev/dsk/*

In the following examples, we use /dev/rdsk/c1t2d0 and /dev/rdsk/c0t2d0, which happen to be the device names for the same disks on both ftsys9 and ftsys10. In the event that the device file names are different on the different nodes, make a careful note of the correspondences.

Creating Physical Volumes

On the configuration node (ftsys9), use the pvcreate command to define disks as physical volumes. This only needs to be done on the configuration node. Use the following commands to create two physical volumes for the sample configuration:

# pvcreate -f /dev/rdsk/c1t2d0 # pvcreate -f /dev/rdsk/c0t2d0

Creating a Volume Group with PVG-Strict Mirroring

Use the following steps to build a volume group on the configuration node (ftsys9). Later, the same volume group will be created on other nodes.

1. First, set up the group directory for vgdatabase:

   # mkdir /dev/vgdatabase

2. Next, create a control file named group in the directory /dev/vgdatabase, as follows:

   # mknod /dev/vgdatabase/group c 64 0xhh0000

   The major number is always 64, and the hexadecimal minor number has the form

   0xhh0000

   where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. Use the following command to display a list of existing volume groups:

   # ls -l /dev/*/group

3. Create the volume group and add physical volumes to it with the following commands:

   # vgcreate -g bus0 /dev/vgdatabase /dev/dsk/c1t2d0 # vgextend -g bus1 /dev/vgdatabase /dev/dsk/c0t2d0

   The first command creates the volume group and adds a physical volume to it in a physical volume group called bus0. The second command adds the second drive to the volume group, locating it in a different physical volume group named bus1. The use of physical volume groups allows the use of PVG-strict mirroring of disks and PV links.

4. Repeat this procedure for additional volume groups.

Building Mirrored Logical Volumes for OPS with LVM Commands

After you create volume groups and define physical volumes for use in them, you define mirrored logical volumes for data, logs, and control files. It is recommended that you use a shell script to issue the commands described in the next sections. The commands you use for creating logical volumes vary slightly depending on whether you are creating logical volumes for OPS redo log files or for use with Oracle data.

Creating Mirrored Logical Volumes for OPS Redo Logs and Control Files

Create logical volumes for use as redo log and control files by selecting mirror consistency recovery. Use the same options as in the following example:

# lvcreate -m 1 -M n -c y -s g -n redo1.log -L 4 /dev/vg_ops

The -m 1 option specifies single mirroring; the -M n option ensures that mirror write cache recovery is set off; the -c y means that mirror consistency recovery is enabled; the -s g means that mirroring is PVG-strict, that is, it occurs between different physical volume groups; the -n redo1.log option lets you specify the name of the logical volume; and the -L 4 option allocates 4 megabytes.

	NOTE: It is important to use the -M n and -c y options for both redo logs and control files. These options allow the redo log files to be resynchronized by SLVM following a system crash before Oracle recovery proceeds. If these options are not set correctly, you may not be able to continue with database recovery.

If the command is successful, the system will display messages like the following:

Logical volume "/dev/vg_ops/redo1.log" has been successfully created with character device "/dev/vg_ops/rredo1.log" Logical volume "/dev/vg_ops/redo1.log" has been successfully extended

Note that the character device file name (also called the raw logical volume name) is used by the Oracle DBA in building the OPS database.

Creating Mirrored Logical Volumes for OPS Data Files

For data files other than the redo logs or control files, choose a mirror consistency policy of "none" by disabling both mirror write caching and mirror consistency recovery. Create logical volumes for use as Oracle data files by using the same options as in the following example:

# lvcreate -m 1 -M n -c n -s g -n system.dbf -L 28 /dev/vg_ops

The -m 1 option specifies single mirroring; the -M n option ensures that mirror write cache recovery is set off; the -c n means that mirror consistency recovery is disabled; the -s g means that mirroring is PVG-strict, that is, it occurs between different physical volume groups; the -n system.dbf option lets you specify the name of the logical volume; and the -L 28 option allocates 28 megabytes.

If the command is successful, the system will display messages like the following:

Logical volume "/dev/vg_ops/system.dbf" has been successfully created with character device "/dev/vg_ops/rsystem.dbf" Logical volume "/dev/vg_ops/system.dbf" has been successfully extended

Note that the character device file name (also called the raw logical volume name) is used by the Oracle DBA in building the OPS database.

Creating OPS Volume Groups on Disk Arrays

The procedure described in this section assumes that you are using RAID-protected disk arrays and LVM's physical volume links (PV links) to define redundant data paths from each node in the cluster to every logical unit on the array.

On your disk arrays, you should use redundant I/O channels from each node, connecting them to separate controllers on the array. Then you can define alternate links to the LUNs or logical disks you have defined on the array. If you are using SAM, choose the type of disk array you wish to configure, and follow the menus to define alternate links. If you are using LVM commands, specify the links on the command line.

The following example shows how to configure alternate links using LVM commands. The following disk configuration is assumed:

8/0.15.0 /dev/dsk/c0t15d0  /* I/O Channel 0 (8/0) SCSI address 15 LUN 0 */
8/0.15.1 /dev/dsk/c0t15d1  /* I/O Channel 0 (8/0) SCSI address 15 LUN 1 */
8/0.15.2 /dev/dsk/c0t15d2  /* I/O Channel 0 (8/0) SCSI address 15 LUN 2 */
8/0.15.3 /dev/dsk/c0t15d3  /* I/O Channel 0 (8/0) SCSI address 15 LUN 3 */
8/0.15.4 /dev/dsk/c0t15d4  /* I/O Channel 0 (8/0) SCSI address 15 LUN 4 */
8/0.15.5 /dev/dsk/c0t15d5  /* I/O Channel 0 (8/0) SCSI address 15 LUN 5 */
 
10/0.3.0 /dev/dsk/c1t3d0   /* I/O Channel 1 (10/0) SCSI address 3 LUN 0 */
10/0.3.1 /dev/dsk/c1t3d1   /* I/O Channel 1 (10/0) SCSI address 3 LUN 1 */
10/0.3.2 /dev/dsk/c1t3d2   /* I/O Channel 1 (10/0) SCSI address 3 LUN 2 */
10/0.3.3 /dev/dsk/c1t3d3   /* I/O Channel 1 (10/0) SCSI address 3 LUN 3 */
10/0.3.4 /dev/dsk/c1t3d4   /* I/O Channel 1 (10/0) SCSI address 3 LUN 4 */
10/0.3.5 /dev/dsk/c1t3d5   /* I/O Channel 1 (10/0) SCSI address 3 LUN 5 */

Assume that the disk array has been configured, and that both the following device files appear for the same LUN (logical disk) when you run the ioscan command:

/dev/dsk/c0t15d0
/dev/dsk/c1t3d0

Use the following procedure to configure a volume group for this logical disk:

LVM will now recognize the I/O channel represented by /dev/dsk/c0t15d0 as the primary link to the disk; if the primary link fails, LVM will automatically switch to the alternate I/O channel represented by /dev/dsk/c1t3d0. Use the vgextend command to add additional disks to the volume group, specifying the appropriate physical volume name for each PV link.

Repeat the entire procedure for each distinct volume group you wish to create. For ease of system administration, you may wish to use different volume groups to separate logs from data and control files.

	NOTE: The default maximum number of volume groups in HP-UX is 10. If you intend to create enough new volume groups that the total exceeds ten, you must increase the maxvgs system parameter and then re-build the HP-UX kernel. In SAM, select the Kernel Configuration area, then choose Configurable Parameters. Maxvgs appears on the list.

Creating Logical Volumes for OPS on Disk Arrays

After you create volume groups and add PV links to them, you define logical volumes for data, logs, and control files. The following are some examples:

# lvcreate -n ops1log1.log -L 4 /dev/vg_ops  
# lvcreate -n opsctl1.ctl -L 4 /dev/vg_ops
# lvcreate -n system.dbf -L 28 /dev/vg_ops
# lvcreate -n opsdata1.dbf -L 1000 /dev/vg_ops

Oracle Demo Database Files

The following set of files is required for the Oracle demo database which you can create during the installation process.

^* The size of the logical volume is larger than the Oracle file size because Oracle needs extra space to allocate a header in addition to the file's actual data capacity.

Create these files if you wish to build the demo database. The three logical volumes at the bottom of the table are included as additional data files, which you can create as needed, supplying the appropriate sizes. If your naming conventions require, you can include the Oracle SID and/or the database name to distinguish files for different instances and different databases. If you are using the ORACLE_BASE directory structure, create symbolic links to the ORACLE_BASE files from the appropriate directory. Example:

# ln -s /dev/vg_ops/ropsctl1.ctl /u01/ORACLE/db001/ctrl01_1.ctl

For more information about Oracle directories, refer to the Oracle Server for HP 9000 Installation and Configuration Guide. More information about the maximum sizes of data files is found in the Oracle 8i Reference (Oracle Part Number A76961-01), Chapter 4, "Database Limits."

After creating these files, set the owner to oracle and the group to dba with a file mode of 660. The logical volumes are now available on the primary node, and the raw logical volume names can now be used by the Oracle DBA.

Displaying the Logical Volume Infrastructure

To display the volume group, use the vgdisplay command:

# vgdisplay -v /dev/vg_ops 

Exporting the Logical Volume Infrastructure

Before the OPS volume groups can be shared, their configuration data must be exported to other nodes in the cluster. This is done either in SAM or by using HP-UX commands, as shown in the following sections.

	NOTE: A volume group using an HP High Availability Disk Array cannot be designated as a cluster lock volume group. You need to use another volume group containing an independent disk for the cluster lock.

Exporting with SAM

In SAM, choose Disks and File Systems, then choose Volume Groups. Open the Actions menu and choose Distribute. Follow the prompts in the dialog box to select the volume group that is to be distributed to one or more additional nodes. Enter the name of each node that is to receive the volume group and select Add. When the list is complete, press OK. SAM automatically configures the volume group for use on the other nodes.

	NOTE: In distributing the volume group, SAM does not import PV links on other nodes. Therefore, to complete the distribution, you use the vgextend command on all nodes to add the alternate PV links to the volume group you have distributed. Use the vgdisplay command to verify the existence of the correct groups of PV links on all nodes.