Cluster-related Volume Manager Utilities and Daemons

vxclustd Daemon

The vxclustd daemon is the Volume Manager cluster reconfiguration daemon. The vxclustd daemon provides communication between the cluster manager and VxVM and initiates cluster reconfigurations. Every node currently in the cluster runs an instance of the vxclustd daemon. Whenever cluster membership changes, the cluster manager notifies the vxclustd daemon, which then initiates a reconfiguration within VxVM.

The vxclustd daemon is started up by the cluster manager when the node initially attempts to join the cluster. The vxclustd daemon first registers with the cluster manager and obtains the following information from the cluster manager:

Registration also provides a callback mechanism for the cluster manager to notify the vxclustd daemon when cluster membership changes. After initializing kernel cluster variables, the vxclustd daemon waits for a callback from the cluster manager. When the vxclustd daemon obtains membership information from the cluster manager, it validates the membership change, and provides the new membership to the kernel. The reconfiguration process continues within the kernel and the vxconfigd daemon. This includes selection of a new master node if necessary, initiation of communication between vxconfigd daemons on the master and slave nodes, and a join protocol at the vxconfigd and kernel levels that validates VxVM objects and distributes VxVM configuration information across the cluster.

If reconfiguration completes successfully, the vxclustd daemon does not take any further action; it waits for the next membership change from the cluster manager. If reconfiguration within the kernel or within the vxconfigd daemon fails, the node must leave the cluster. The kernel fails I/Os in progress to shared disks, and stops access to shared disks and the vxclustd daemon. The vxclustd daemon invokes the cluster manager command to halt the cluster on this node.

When a clean node shutdown is performed, the vxclustd daemon waits until kernel cluster reconfiguration completes and then exits.

vxconfigd Daemon

The vxconfigd daemon is the Volume Manager configuration daemon. The vxconfigd daemon maintains configurations of VxVM objects. The vxconfigd daemon receives cluster-related instructions from the kernel. A separate copy of the vxconfigd daemon resides on each node; these copies communicate with each other through networking facilities. For each node in a cluster, Volume Manager utilities communicate with the vxconfigd daemon running on that particular node; utilities do not attempt to connect with vxconfigd daemons on other nodes. During startup of the cluster, kernel tells the vxconfigd daemon to begin cluster operation and tells it whether it is a master or slave node.

When a node is initialized for cluster operation, the kernel tells the vxconfigd daemon that the node is about to join the cluster and provides the vxconfigd daemon with the following information (from the cluster manager configuration database):

On the master node, the vxconfigd daemon sets up the shared configuration (i.e., imports the shared disk groups) and informs the vxclustd daemon when it is ready for slaves to join.

On slave nodes, the kernel tells the vxconfigd daemon when the slave node can join the cluster. When the slave node joins the cluster, the vxconfigd daemon and the Volume Manager kernel communicate with their counterparts on the master in order to set up the shared configuration.

When a node leaves the cluster, the vxconfigd daemon notifies the kernel on all the other nodes. The master node then performs any necessary cleanup. If the master node leaves the cluster, the kernels choose a new master node and the vxconfigd daemons on all nodes are notified of the choice.

The vxconfigd daemon also participates in volume reconfiguration. See “Volume Reconfiguration” for information on the role of the vxconfigd daemon in volume reconfiguration.

vxconfigd Daemon Recovery

The Volume Manager vxconfigd daemon can be stopped and/or restarted at any time. While the vxconfigd daemon is stopped, volume reconfigurations cannot take place and other nodes cannot join the cluster until the vxconfigd daemon is restarted. In the cluster, the vxconfigd daemons on the slaves are always connected to the vxconfigd daemon on the master. It is therefore not advisable to stop the vxconfigd daemon on any clustered node.

If the vxconfigd daemon is stopped, different actions are taken depending on which node has a stopped daemon:

• If the vxconfigd daemon is stopped on the slave(s), the master takes no action. When the vxconfigd daemon is restarted on the slave, the slave's vxconfigd daemon attempts to reconnect to the master's and re-acquire the information about the shared configuration. (The kernel's view of the shared configuration is unaffected, and so is access to the shared disks.) Until the slave vxconfigd daemon has successfully rejoined to the master, it has very little information about the shared configuration and any attempts to display or modify the shared configuration can fail. In particular, if the shared disk groups are listed (using the vxdg list command), they are marked as disabled; when the rejoin completes successfully, they are marked as enabled.

• If the vxconfigd daemon is stopped on the master, the vxconfigd daemon on the slave(s) attempts to rejoin to the master periodically. This is not successful until the vxconfigd daemon is restarted on the master. In this case, the slave vxconfigd daemon information about the shared configuration has not been lost, so configuration displays are accurate.

• If the vxconfigd daemon is stopped on both the master and the slave(s), the slave does not display accurate configuration information until the vxconfigd daemon is restarted on both nodes and they reconnect.

When the vxclustd daemon notices that the vxconfigd daemon is stopped on a node, the vxclustd daemon restarts the vxconfigd daemon.

	NOTE: With VxVM, the -r reset option to the vxconfigd daemon restarts the vxconfigd daemon and creates all states from scratch. This option is not available while a node is in the cluster because it causes the loss of cluster information; if this option is used under these circumstances, the vxconfigd daemon does not start.

vxdctl Utility

When the vxdctl utility is executed as "vxdctlenable", if DMP identifies a DISABLED primary path of a shared disk in an active/passive type disk array as physically accessible, this path is marked as ENABLED. However, I/Os continue to use the current path and are not routed through the path that has been marked ENABLED.

This is a deviation from single host behavior of this command where I/Os automatically failback to the primary path.

The vxdctl utility manages some aspects of the vxconfigd volume configuration daemon.The -c option can be used to request cluster information. To use the vxdctl utility to determine whether the vxconfigd daemon is enabled and/or running, use the following command:

# vxdctl -c mode

Depending on the circumstances, this produces output similar to the following:

mode: enabled: cluster active - MASTER
mode: enabled: cluster active - SLAVE
mode: enabled: cluster inactive
mode: enabled: cluster active - role not set

	NOTE: If the vxconfigd daemon is disabled, no cluster information is displayed.

The vxdctl utility lists the cluster protocol version and cluster protocol range. After all the nodes in the cluster are updated with the new cluster protocol, update the entire cluster using the following command:

# vxdctl upgrade

Check the existing cluster protocol version using the following command:

# vxdctl protocolversion
Cluster running at protocol 10

Display the maximum and minimum cluster protocol version supported by the current VxVM release using the following command:

# vxdctl protocolrange
minprotoversion: 10, maxprotoversion: 20

The vxdctl list command displays the cluster protocol version running on a node. The vxdctl list command produces the following output:

Volboot file
version: 3/1
seqno: 0.19
cluster protocol version: 20
hostid: giga
entries:

The vxdctl support command displays the maximum and minimum protocol version supported by the node and the current protocol version. The following is the output of the vxdctl support command:

Support information:
  vold_vrsn: 11
  dg_minimum: 60
  dg_maximum: 70
  kernel: 10
  protocol_minimum: 10
  protocol_maximum: 20
  protocol_current: 20

vxdg Utility

The vxdg utility manages Volume Manager disk groups. The vxdg utility can be used to specify that a disk group is cluster-shareable. The -s option to the vxdg utility is provided to initialize or import a disk group as "shared."

If the cluster software has been run to set up the cluster, a shared disk group can be created using the following command:

# vxdg -s init diskgroup [medianame=]accessname

where:

Importing disk groups

Disk groups can be imported as shared using the vxdg -s import command. If the disk groups were set up before the cluster software was run, the disk groups can be imported into the cluster arrangement using the following command:

# vxdg -s import diskgroup

where diskgroup is the disk group name or ID. On subsequent cluster restarts, the disk group is automatically imported as shared. Note that it can be necessary to deport the disk group (using the vxdg deport diskgroup command) before invoking this command.

Converting a disk group from shared to private

A disk group can be converted from shared to private by deporting it via vxdg deport and then importing it with the vxdg import diskgroup command.

	NOTE: The system cannot tell if a disk is shared. To protect data integrity when dealing with disks that can be accessed by multiple systems, the system administrator must be careful to use the correct designation when adding a disk to a disk group. If the administrator attempts to add a disk that is not physically shared to a shared disk group, the Volume Manager allows this on the node where the disk is accessible if that node is the only one in the cluster. However, other nodes are unable to join the cluster. Furthermore, if the administrator attempts to add the same disk to different disk groups on two nodes at the same time, the results are undefined. All configurations should therefore be handled on one node only.

Force-importing a disk group or force-adding a disk

The vxdg command has a force option (-f) that can be used to force-import a disk group or force-add a disk to a disk group.

	NOTE: The force option(-f) should be used with caution and should only be used if the system administrator is fully aware of the possible consequences.

When a cluster is restarted, VxVM may refuse to auto-import a disk group for one of the following reasons:

• A disk in that disk group is no longer accessible because of hardware errors on the disk. In this case, the system administrator can reimport the disk group with the force option using the following command:

  # vxdg -s -f import diskgroup

• Some of the nodes to which disks in the disk group are attached are not currently in the cluster, so the disk group cannot access all of its disks. In this case, a forced import is unsafe and should not be attempted (because it can result in inconsistent mirrors).

If VxVM does not add a disk to an existing disk group (because that disk is not attached to the same node(s) as the other disks in the disk group), the system administrator can force-add the disk using the following command:

# vxdg -f adddisk -g diskgroup [medianame=]accessname

Activating a shared disk group

A shared disk group can be activated using the following command:

# vxdg -g diskgroup set activation=mode

where mode is one of the following:

• exclusivewrite

• sharedwrite

• readonly

• sharedread

• off

Listing shared disk groups

vxdg can also be used to list shared disk groups. To display one line of information for each disk group, use the following command:

# vxdg list

The output from this command is as follows:

NAME STATE ID rootdg enabled 774215886.1025.teal group2 enabled,shared 774575420.1170.teal group1 enabled,shared 774222028.1090.teal

Shared disk groups are designated with the flag shared.

To display one line of information for each shared disk group, use the following command:

# vxdg -s list

The output for this command is as follows:

NAME STATE ID group2 enabled,shared 774575420.1170.teal group1 enabled,shared 774222028.1090.teal

To display information about one specific disk group, including whether it is shared or not, use the following command:

# vxdg list diskgroup

where diskgroup is the disk group name.

For example, the output for vxdg list group1 on the master (for the disk group group1) is as follows:

Group: group1 dgid: 774222028.1090.teal import-id: 32768.1749 flags: shared version: 70 activation: exclusive-write detach-policy: local copies: nconfig=default nlog=default config: seqno=0.1976 permlen=1456 free=1448 templen=6 loglen=220 config disk c1t0d0s2 copy 1 len=1456 state=clean online config disk c1t1d0s2 copy 1 len=1456 state=clean online log disk c1t0d0s2 copy 1 len=220 log disk c1t1d0s2 copy 1 len=220

Note that the flags: field is set to shared. The output for the same command is slightly different on a slave.

vxdisk Utility

The vxdisk utility manages Volume Manager disks. To use the vxdisk utility to determine whether a disk is part of a cluster-shareable disk group, use the following command:

# vxdisk list accessname

where accessname is the disk access name (or device name).

The output from this command (for the device c4t1d0) is as follows:

Device:    c4t1d0
devicetag: c4t1d0
type:      simple
hostid:    hpvm2
disk:      name=disk01 id=963616090.1034.hpvm2
timeout:   30
group:     name=rootdg id=963616065.1032.hpvm2
flags:     online ready private autoconfig autoimport imported
pubpaths:  block=/dev/vx/dmp/c4t1d0 char=/dev/vx/rdmp/c4t1d0
version:   2.1
iosize:    min=1024 (bytes) max=64 (blocks)
public:    slice=0 offset=1152 len=8612361
private:   slice=0 offset=128 len=1024
update:    time=964035962 seqno=0.30
headers:   0 248
configs:   count=1 len=727
logs:      count=1 len=110
Defined regions:
 config   priv 000017-000247[000231]: copy=01 offset=000000 enabled
 config   priv 000249-000744[000496]: copy=01 offset=000231 enabled
 log      priv 000745-000854[000110]: copy=01 offset=000000 enabled
 lockrgn  priv 000855-000919[000065]: part=00 offset=000000
Multipathing information:
numpaths:   2
c4t1d0  state=enabled   type=secondary
c5t3d0  state=enabled   type=primary

Note that the clusterid: field is set to cvm (the name of the cluster) and the flags: field includes an entry for shared. When a node is not joined, the flags: field contains the autoimport flag instead of imported.

vxdmpadm Utility

The vxdmpadm utility is a command line administrative interface for the Dynamic Multipathing feature of VxVM. This utility can be used to list DMP database information and perform other system administration related tasks. For more information, see the manual page vxdmpadm(1M) and “Disabling and Enabling Controllers”.

Enabling and Disabling Controllers

Volume Manager does not allow enabling or disabling of controllers connected to a disk that is part of a shared VxVM disk group.

For example, consider a Galaxy disk array that is connected through controller c0 to a host. This controller has a disk that is part of a shared disk group. In such a situation, following operations fail on that host:

# vxdmpadm disable ctlr=c0 # vxdmpadm enable ctlr=c0

The following error message is displayed:

vxvm: vxdmpadm: ERROR: operation not supported for shared disk arrays.

vxrecover Utility

The vxrecover utility recovers plexes and volumes after disk replacement.

When a node leaves the cluster, it can leave some mirrors in an inconsistent state. The vxrecover utility performs recovery on all volumes in this state. The -c option causes the vxrecover utility to perform recovery for all volumes in cluster-shareable disk groups.The vxclustd daemon automatically calls the vxrecover -c utility, when necessary.

	NOTE: While the vxrecover utility is active, there may be some degradation in system performance.

vxstat Utility

The vxstat utility returns statistics for specified objects. In a cluster environment, the vxstat utility gathers statistics from all of the nodes in the cluster. The statistics give the total usage, by all nodes, for the requested objects. If a local object is specified, its local usage is returned.

The caller can optionally specify a subset of nodes using the following command:

# vxstat -g diskgroup -n node[,node...]

where node is an integer. If a comma-separated list of nodes is supplied, the vxstat utility displays the sum of the statistics for the nodes in the list.

For example, to obtain statistics for node 2, volume vol1,use the following command:

# vxstat -g group1 -n 2 vol1

This command produces output similar to the following:

                    OPERATIONS           BLOCKS         AVG TIME(ms)
TYP  NAME          READ     WRITE      READ     WRITE    READ  WRITE
vol  vol1          2421         0    600000         0    99.0    0.0

To obtain and display statistics for the entire cluster, use the following command:

# vxstat -b

The statistics for all nodes are added together. For example, if node 1 did 100 I/Os and node 2 did 200 I/Os, the vxstat -b command returns 300 I/Os.