 |
» |
|
|
 |
You can check status using Serviceguard Manager or on a cluster
node’s command line. Reviewing
Cluster and Package Status with Serviceguard Manager |  |
Serviceguard Manager shows status several ways. On the map, cluster object icons have
borders to show problems. To the right of the icons, a badge gives
information about the type of problem. When you hover the mouse,
a popup gives you information. There are more details in the cluster, node, and
package property sheets. Cluster multi-node packages’ properties
are contained in the cluster properties.
Reviewing
Cluster and Package States with the cmviewcl Command |  |
Information about cluster
status is stored in the status database, which is maintained on
each individual node in the cluster. You can display information
contained in this database by issuing the cmviewcl command:# cmviewcl -v You can issue the cmviewcl command with non-root access. To allow access, clusters
with Serviceguard version A.11.16 or later grant access by adding
a Monitor role in the cluster configuration file. Earlier versions
allow access by adding a pair (<nodename> <nonrootuser>)
to their cmclnodelist file. The cmviewcl command, when issued with the -v option, displays information about all the nodes
and packages in a running cluster, together with the settings of
parameters that determine failover behavior.  |  |  |  |  | TIP: Some commands take longer to complete in large configurations.
In particular, you can expect Serviceguard’s CPU usage
to increase during cmviewcl -v as the number of packages and services increases. |  |  |  |  |
You can also specify that the output should be formatted as
it was in a specific earlier release by using the -r option indicating
the release format you wish. Example: # cmviewcl -r A.11.09 The formatting options lets you choose a style. The tabulated
format is designed for viewing. The line format is designed for
scripting, and is easily parsed. See the man page for a detailed description of other cmviewcl options. Viewing
multi-node Information To see multi-node
package configuration information, status, and dependencies in a
CFS cluster, you can use the cfs commands, such as cfsdgadm show_package <diskgroup>, cfsmntadm show_package <mountpoint>, getconf -p <mnpkg> | grep DEPENDENCY. The cmviewcl -v command output lists dependencies throughout the cluster.
For a specific package’s dependencies, use the -p pkgname option. Types
of Cluster and Package States A cluster or its component nodes may be in several different
states at different points in time. The following sections describe
many of the common conditions the cluster or package may be in. The status of a cluster may be one of
the following: Up. At least one node has a running
cluster daemon, and reconfiguration is not taking place. Down. No cluster daemons are running on any cluster
node. Starting. The cluster is in the process of determining
its active membership. At least one cluster daemon is running. Unknown. The node on which the cmviewcl command is issued cannot communicate with other nodes
in the cluster.
The status of a node is either up (active as
a member of the cluster) or down (inactive in
the cluster), depending on whether its cluster daemon is running
or not. Note that a node might be down from the cluster perspective,
but still up and running HP-UX. A node may also be in one of the following states: Failed. A node never
sees itself in this state. Other active members of the cluster will
see a node in this state if that node was in an active cluster,
but is no longer, and is not halted. Reforming. A node is in this state
when the cluster is re-forming. The node is currently running the
protocols which ensure that all nodes agree to the new membership
of an active cluster. If agreement is reached, the status database
is updated to reflect the new cluster membership. Running. A node in this state has
completed all required activity for the last re-formation and is
operating normally. Halted. A node never sees itself
in this state. Other nodes will see it in this state after the node
has gracefully left the active cluster, for instance with a cmhaltnode command. Unknown. A node never sees itself
in this state. Other nodes assign a node this state if it has never
been an active cluster member.
The status of a
package can be one of the following: Up. The package control
script is active. Down. The package control script
is not active. Unknown. Serviceguard cannot determine
the status at this time.
A system multi-node package is up when it is running on all the
active cluster nodes. A multi-node package is up if it is running
on any of its configured nodes. The state of the
package can be one of the following: Starting. The start
instructions in the control script are being run. Running. Services are active and
being monitored. Halting. The halt instructions
in the control script are being run.
Package
Switching Attributes Packages have the following switching attributes: Auto Run. For failover packages, enabled
means that Serviceguard can switch the package to another node in
the event of failure. For system multi-node packages, Enabled means an instance
of the package can start on a new node joining the cluster (Disabled
means it will not). Switching Enabled for a Node. For failover packages,
enabled means that the package can switch to the referenced node.
Disabled means that the package cannot switch to the specified node
until the node is enabled for the package using the cmmodpkg command. Every failover package is marked Enabled or Disabled for each
node that is either a primary or adoptive node for the package. For multi-node packages, node switching Disabled means the package
cannot start on that node.
Services have only status, as follows: Up. The service is
being monitored. Down. The service is not running.
It may have halted or failed. Uninitialized. The service is included
in the cluster configuration, but it was not started with a run
command in the control script.
The network interfaces have only status, as follows: Unknown. Serviceguard cannot determine
whether the interface is up or down. This can happen when the cluster
is down. A standby interface has this status.
The serial line has only status, as follows: Up. Heartbeats are
received over the serial line. Down. Heartbeat has not been received
over the serial line within 2 times the NODE_TIMEOUT value. Recovering. A corrupt message was
received on the serial line, and the line is in the process of resynchronizing. Unknown. We cannot determine whether
the serial line is up or down. This can happen when the remote node
is down.
Failover
and Failback PoliciesFailover packages can be configured with one of two values
for the FAILOVER_POLICY parameter: CONFIGURED_NODE. The package fails over to the next node in the node
list in the package configuration file. MIN_PACKAGE_NODE. The package fails over to the node in the cluster with
the fewest running packages on it.
Failover packages can also be configured with one of two values
for the FAILBACK_POLICY parameter: AUTOMATIC. With this setting, a package, following a failover,
returns to its primary node when the primary node becomes available
again. MANUAL. With this setting, a package, following a failover,
must be moved back to its original node by a system administrator.
Failover and failback policies are displayed in the output
of the cmviewcl -v command. Examples
of Cluster and Package States The following sample output from the cmviewcl -v command shows status for the cluster in the sample
configuration. Everything is running normally; both nodes in the cluster
are running, and the packages are in their primary locations.  |
CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 56/36.1 lan0 STANDBY up 60/6 lan1 PACKAGE STATUS STATE AUTO_RUN NODE pkg1 up running enabled ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 service1 Subnet up 0 0 15.13.168.0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled ftsys9 (current) Alternate up enabled ftsys10 NODE STATUS STATE ftsys10 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 28.1 lan0 STANDBY up 32.1 lan1 PACKAGE STATUS STATE AUTO_RUN NODE pkg2 up running enabled ftsys10 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 service2 Subnet up 0 0 15.13.168.0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled ftsys10 (current) Alternate up enabled ftsys9
|
 |
If the cluster is using a quorum server for tie-breaking services,
the display shows the server name, state and status following the
entry for each node, as in the following excerpt from the output
of cmviewcl -v: CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running Quorum Server Status: NAME STATUS STATE lp-qs up running ... NODE STATUS STATE ftsys10 up running Quorum Server Status: NAME STATUS STATE lp-qs up running |
If the cluster is using the VERITAS Cluster Volume Manager,
version 3.5, for disk storage, the system multi-node package VxVM-CVM-pkg must be running on all active nodes for applications
to be able to access CVM disk groups. The system multi-node package
is named SG-CFS-pkg if the cluster is using version 4.1 of the VERITAS Cluster
Volume Manager. The VxVM-CVM-pkg package is shown in the following output of the cmviewcl command: CLUSTER STATUS example up NODE STATUS STATE ftsys7 down halted ftsys8 down halted ftsys9 up running ftsys10 up running MULTI_NODE_PACKAGES: PACKAGE STATUS STATE AUTO_RUN SYSTEM VxVM-CVM-pkg up running enabled yes |
When you use the -v option, the display shows the system multi-node package
associated with each active node in the cluster, as in the following: MULTI_NODE_PACKAGES: PACKAGE STATUS STATE AUTO_RUN SYSTEM VxVM-CVM-pkg up running enabled yes NODE STATUS SWITCHING ftsys7 down disabled NODE STATUS SWITHCHING ftsys8 down disabled NODE STATUS SWITCHING ftsys9 up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 VxVM-CVM-pkg.srv NODE STATUS SWITCHING ftsys10 up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 VxVM-CVM-pkg.srv |
If the cluster is using the VERITAS Cluster File System, the
system multi-node package SG-CFS-pkg must be running on all active nodes, and the multi-node
packages for disk group and mount point must also be running on
at least one of their configured nodes. The following shows an example of cmviewcl output for status
of these packages:
# cmviewcl -v -p SG-CFS-pkg MULTI_NODE_PACKAGES PACKAGE STATUS STATE AUTO_RUN SYSTEM SG-CFS-pkg up running enabled yes NODE_NAME STATUS SWITCHING ftsys9 up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service up 0 0 SG-CFS-cmvxpingd NODE_NAME STATUS SWITCHING ftsys10 up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service up 0 0 SG-CFS-cmvxpingd |
Status
After Halting a PackageAfter halting the failover package pkg2 with the cmhaltpkg command, the output of cmviewcl-v is as follows:  |
CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 56/36.1 lan0 STANDBY up 60/6 lan1 PACKAGE STATUS STATE AUTO_RUN NODE pkg1 up running enabled ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 service1 Subnet up 15.13.168.0 Resource up /example/float Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled ftsys9 (current) Alternate up enabled ftsys10 NODE STATUS STATE ftsys10 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 28.1 lan0 STANDBY up 32.1 lan1 UNOWNED_PACKAGES PACKAGE STATUS STATE AUTO_RUN NODE pkg2 down halted disabled unowned Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS NODE_NAME NAME Resource up ftsys9 /example/float Subnet up ftsys9 15.13.168.0 Resource up ftsys10 /example/float Subnet up ftsys10 15.13.168.0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled ftsys10 Alternate up enabled ftsys9
|
 |
Pkg2 now has the status “down”, and it
is shown as in the unowned state, with package switching disabled.
Resource “/example/float,” which is configured
as a dependency of pkg2, is down on one node. Note that switching
is enabled for both nodes, however. This means that once global
switching is re-enabled for the package, it will attempt to start
up on the primary node. Status
After Moving the Package to Another NodeAfter issuing the following command: # cmrunpkg -n ftsys9 pkg2 |
the output of the cmviewcl -v command is as follows:  |
CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 56/36.1 lan0 STANDBY up 60/6 lan1 PACKAGE STATUS STATE AUTO_RUN NODE pkg1 up running enabled ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 service1 Subnet up 15.13.168.0 Resource up /example/float Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled ftsys9 (current) Alternate up enabled ftsys10 PACKAGE STATUS STATE AUTO_RUN NODE pkg2 up running disabled ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 service2.1 Subnet up 15.13.168.0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled ftsys10 Alternate up enabled ftsys9 (current) NODE STATUS STATE ftsys10 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 28.1 lan0 STANDBY up 32.1 lan1
|
 |
Now pkg2 is running on node ftsys9. Note that it is still
disabled from switching. Status
After Auto Run is EnabledThe following command changes package switching flag back
to Auto Run Enabled: The output of the cmviewcl command is now as follows: CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running PACKAGE STATUS STATE AUTO_RUN NODE pkg1 up running enabled ftsys9 pkg2 up running enabled ftsys9 NODE STATUS STATE ftsys10 up running |
Both packages are now running on ftsys9 and pkg2 is enabled
for switching. Ftsys10 is running the daemon and no packages are
running on ftsys10. Status
After Halting a NodeAfter halting ftsys10, with the following command: the output of cmviewcl is as follows on ftsys9: CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running PACKAGE STATUS STATE AUTO_RUN NODE pkg1 up running enabled ftsys9 pkg2 up running enabled ftsys9 NODE STATUS STATE ftsys10 down halted |
This output is seen on both ftsys9 and ftsys10. If you are using a serial (RS232) line as a heartbeat connection,
you will see a list of configured RS232 device files in the output
of the cmviewcl -v command. The following shows normal running status: CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 56/36.1 lan0 Serial_Heartbeat: DEVICE_FILE_NAME STATUS CONNECTED_TO: /dev/tty0p0 up ftsys10 /dev/tty0p0 NODE STATUS STATE ftsys10 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 28.1 lan0 Serial_Heartbeat: DEVICE_FILE_NAME STATUS CONNECTED_TO: /dev/tty0p0 up ftsys9 /dev/tty0p0 |
The following display shows status after node ftsys10 has
halted: CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 56/36.1 lan0 Serial_Heartbeat: DEVICE_FILE_NAME STATUS CONNECTED_TO: /dev/tty0p0 unknown ftsys10 /dev/tty0p0 NODE STATUS STATE ftsys10 down running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 28.1 lan0 Serial_Heartbeat: DEVICE_FILE_NAME STATUS CONNECTED_TO: /dev/tty0p0 unknown ftsys9 /dev/tty0p0 |
The following shows status when the serial line is not working: CLUSTER STATUS example up NODE STATUS STATE ftsys9 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 56/36.1 lan0 Serial_Heartbeat: DEVICE_FILE_NAME STATUS CONNECTED_TO: /dev/tty0p0 down ftsys10 /dev/tty0p0 NODE STATUS STATE ftsys10 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 28.1 lan0 Serial_Heartbeat: DEVICE_FILE_NAME STATUS CONNECTED_TO: /dev/tty0p0 down ftsys9 /dev/tty0p0 |
Viewing
Data on Unowned PackagesThe following example shows packages that are currently unowned,
that is, not running on any configured node. Information on monitored
resources is provided for each node on which the package can run;
this allows you to identify the cause of a failure and decide where
to start the package up again. UNOWNED_PACKAGES PACKAGE STATUS STATE AUTO_RUN NODE PKG3 down halted disabled unowned Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover min_package_node Failback automatic Script_Parameters: ITEM STATUS NODE_NAME NAME Resource up manx /resource/random Subnet up manx 192.8.15.0 Resource up burmese /resource/random Subnet up burmese 192.8.15.0 Resource up tabby /resource/random Subnet up tabby 192.8.15.0 Resource up persian /resource/random Subnet up persian 192.8.15.0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled manx Alternate up enabled burmese Alternate up enabled tabby Alternate up enabled persian
|
Viewing
Data on System multi-node packagesThe following example shows a cluster that includes system
multi-node packages as well as standard Serviceguard packages. The
system multi-node packages are running on all nodes in the cluster,
whereas the standard packages run on only one node at a time. CLUSTER STATUS cats up NODE STATUS STATE manx up running PACKAGE STATUS STATE AUTO_RUN NODE pkg1 up running enabled manx NODE STATUS STATE tabby up running PACKAGE STATUS STATE AUTO_RUN NODE pkg2 up running enabled tabby SYSTEM_MULTI_NODE_PACKAGES: PACKAGE STATUS STATE VxVM-CVM-pkg up running
|
Checking
Status with Cluster File System If the cluster his using the cluster file system, you can
check status with the cfscluster command, as show in the example below: #cfscluster status Node : ftsys9 Cluster Manager : up CVM state : up (MASTER) MOUNT POINT TYPE SHARED VOLUME DISK GROUP STATUS /var/opt/sgtest/ tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/1vol1 regular lvol1 vg_for_cvm_dd5 MOUNTED /var/opt/sgtest/ tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/lvol4 regular lvol4 vg_for_cvm_dd5 MOUNTED Node : ftsys8 Cluster Manager : up CVM state : up MOUNT POINT TYPE SHARED VOLUME DISK GROUP STATUS /var/opt/sgtest/ tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/lvol1 regular lvol1 vg_for_cvm_veggie_dd5 MOUNTED /var/opt/sgtest/ tmp/mnt/dev/vx/dsk/ vg_for_cvm1_dd5/lvol4 regular lvol4 vg_for_cvm_dd5 MOUNTED
|
Status
of the packages with a Cluster File System installed You can use cmviewcl to see the status of the package and the cluster file system
on all nodes, as show in the example below: #cmviewcl -v -p SG-CFS-pkg MULTI_NODE_PACKAGES PACKAGE STATUS STATE AUTO_RUN SYSTEM SG-CFS-pkg up running enabled yes NODE_NAME STATUS SWITCHING soy up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service up 0 0 SG-CFS-cmvxpingd NODE_NAME STATUS SWITCHING tofu up enabled Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 SG-CFS-vxconfigd Service up 5 0 SG-CFS-sgcvmd Service up 5 0 SG-CFS-vxfsckd Service up 0 0 SG-CFS-cmvxd Service up 0 0 SG-CFS-cmvxpingd
|
Status
of CFS disk group packagesTo see the status of the disk group, use the cfsdgadm display command. For example, for the diskgroup logdata, enter: # cfsdgadm display -v logdata NODE NAME ACTIVATION MODE ftsys9 sw (sw) MOUNT POINT SHARED VOLUME TYPE ftsys10 sw (sw) MOUNT POINT SHARED VOLUME TYPE ... To see which package is monitoring a disk group, use the cfsdgadm show_package command. For example, for the diskgroup logdata, enter: # cfsdgadm show_package logdata SG-CFS-DG-1
Status
of CFS mount point packagesTo see the status of the mount point package, use the cfsmntadm display command. For example, for the mount point /tmp/logdata/log_files, enter: # cfsmntadm display -v /tmp/logdata/log_files Mount Point : /tmp/logdata/log_files Shared Volume : lvol1 Disk Group : logdata To see which package is monitoring a mount point, use the cfsmntadm show_package command. For example, for the diskgroup logdata: # cfsmntadm show_package /tmp/logdata/log_files SG-CFS_MP-1
|