Logging and Tracing

Use logging for most troubleshooting activities. By default the monitors log to api.log. Logging to /var/adm/syslog/syslog.log is ON by default for the disk monitor and OFF by default for the remaining monitors. Tracing should only be used when instructed to do so by HP support personnel. This is not available with all monitors.

EMS Logging

Log files in /etc/opt/resmon/log/ contain information logged by the monitors.

If you are having a problem with interfaces to EMS or MC/ServiceGuard, look at the client.log. With the default level of logging, only audit and error messages are logged. An example of an audit message is:

User event occurred at Thu Jul 31 16:13:31 1997 Process ID: 10404 (client) Log Level: Audit + /vg/vg00/lv/copies/* (8 instances) If (<1), OpC (m/n), 18000s, Thu Jul 31 16:13:31 1997

Plus (+) indicates that a request has been added. Minus (-) indicates that a request has been deleted. A minus (-) followed by a plus (+) indicates a modification. Events sent to targets are marked with a period (.). Errors are marked with Log Level: Error or with Log Level: Warning.

Look at the api.log if you seem to be having a problem with a specific monitor. Check for warnings or errors.

Some monitors have their own logs, refer to the manpage for individual monitors.

Log File Size

EMS log files are normally under /etc/opt/resmon/log. EMS requires 4 MB disk space in the /etc/opt directory for log files - api.log, registrar.log, client.log, emsagent.log, api.log.old, registrar.log.old, client.log.old, and emsagent.log.old. An additional 2 MB disk space is required for reslog.html, and reslog.old.html for archiving monitor data. Apart from 6 MB monitor disk space, you can reserve additional disk space for EMS monitors logging. To relocate the directory, enter the following commands:

mkdir /newpath/resmon mv /etc/opt/resmon/log/newpath/resmon # create /newpath/resmon/log # remove /etc/opt/resmon/log ln -s /newpath/resmon/log /etc/opt/resmon/log




	NOTE: EMS requires that `/etc/opt/resmon`, the parent directory, reside on the root file system. Do not move all of `/etc/opt/resmon` to another file system.

EMS Log Files

EMS log files are stored in the following locations:

/etc/opt/resmon/log
/var/opt/resmon/log

The following log files are stored in the /etc/opt/resmon/log path:

api.log
This file contains logs of the processes that make use of the EMS API to interact with EMS. All EMS Monitors such as EMS HA Monitors (diskmond, mibmond etc.), EMS Hardware Monitors (disk_em, ha_disk_array, dm_stape etc.), EMS Kernel Resource Monitor (krmond) and the Peripheral Status Monitor (psmmon) log to this file.
Once the file reaches 500KB in size, it is automatically copied to api.log.old and the previous api.log.old file is lost.
client.log
This file contains the logs of all EMS Clients. There is a number of possible client processes such as the Persistence Client (p_client), the EMS GUI (invoked by HP SMH), the Peripheral Status Monitor Client/Target daemon (psmctd), monconfig, resls and other.Once the file reaches 500KB in size, it is automatically copied to client.log.old and the previous client.log.old file is lost.
registrar.log
All data logged by the registrar processes is stored in this file. Once the file reaches 500KB in size, it is automatically copied to registrar.log.old and the previous registrar.log.old file is lost.
emsagent.log
The EMS subagent, emsagent, logs errors into the /etc/opt/resmon/log/emsagent.log file.
reslog.html
Monitor data is archived in the file /etc/opt/resmon/log/reslog.html. All monitors write to the same file. Once the file reaches 500KB in size, it is automatically copied to /etc/opt/resmon/log/reslog.old.html. But, if the /etc/opt/resmon/unlimited_log file exists, the reslog.html file will continue to grow in size.

The following log files are stored in the /var/opt/resmon/log path:

mibmond.log
Logfile of the EMS HA Monitors mibmond, lanmond, fsmond, pkgmond, svcmond and clustermond.
diskmond.log
Logfile of the EMS HA Monitor, diskmond, for LVM disks.
rdbmsmond.log
Logfile of the EMS HA Monitor, rdbmsmond, for databases.

Debug logging of the EMS framework

To enable debug logging of the EMS framework, i.e. of the EMS clients, the EMS registrar and of the EMS API functions, one has to create the file /etc/opt/resmon/debug.

# touch /etc/opt/resmon/debug

Debug logging output will be available in /etc/opt/resmon/log when a system is rebooted or when EMS is restarted. To restart EMS, execute the following steps:

Kill all EMS monitors.
Stop EMS clients.
Kill all registrar processes.
Kill p_client.

The p_client process will be restarted immediately. EMS monitor will be started if the persistent requests are present. The non persistent requests will be lost.

Using resls to Check EMS Communication

resls is a client application that can verify that the communication between client, registrar and a monitor works. If you are dealing with a problem that seems to be related to some sort of communication issue between EMS processes, use resls to verify whether all monitors are affected or only some.

For example:

If you type the following command at the HP-UX prompt, 1)

# resls /system/numUsers

the above command fails to check if any request is available for a required resource. The resources managed by the mibmond can be listed with resls /system command. You should also verify if another resource of the same monitor fails, for example, mibmond.

Another mibmond resource: 2)

# resls /system/jobQueue1Min

A diskmond resource: 3)

# resls /vg/vg00/pv_summary

If 2) fails but 3) succeeds, it is likely that the problem is with mibmond or with the SNMP setup, where the mibmond monitor gets its data.
If 2) and 3) fail, it is likely that the problem is with EMS framework, e.g. the communication of the resls(1m) client with the registrar fails.
If 2) succeeds, the problem is most likely with the specific resource "/system/numUsers", since the query of the other resource of the same EMS monitor works fine.

Using resls -s to Check the Status of an EMS Resource

From EMS A.03.20 onwards, you can view the current value of the resource instance by using the new "-s" option with resls, e.g.

# resls -s /vg/vgomni/pv_summary

COntacting Registrar on grcdg236
NAME: /vg/vgomni/pv_summary
DESCRIPTION: Physical Volume and PVLinks Summary
(/vg/<vgName>/pvsummary)
[...]

There are 4 valid states reported for /vg/vgomni/pv_summary:State Value State Name------------------------

1 UP
2 PVG_UP
3 SUSPECT
4 DOWN

EMS Persistence Files

The EMS persistence files are located under/etc/opt/resmon/persistence. They contain the monitoring requests that are persistent, that is reconfigured by p_client when the monitor dies or after a system reboot. For example, all monitor requests configured in HP SMH, through the EMS GUI or the EMS CLI, are persistent and are stored in the m.xxxxxxx file under /etc/opt/resmon/persistence.

The EMS resource monitoring requests for MC/ServiceGuard are defined as package resource dependencies. These monitoring requests are not persistent requests, that is, if the monitor were to die, the p_client will not restart the monitor and reissue the monitoring requests. This is left to the cluster daemon (cmcld) to manage. MC/ServiceGuard has therefore its own persistence database.

The persistence file names are normally having the format m.<number>, where <number> is created by using a hash algorithm on the EMS monitor start-up string stored in its dictionary file in /etc/opt/resmon/dictionary. If the invocation command changes (e.g. by adding command line options), the persistence file name also changes.

Structure of the Persistence File

The first 8 bytes give the length of monitor request: 0000014a, always followed by entries of the form "IIII,LLLL,DDDD;" where

I represents the ID of the field.
L represents the length of the data field.
D represents the data.

Example:

2019,4,1002;2019: ID is RMObjectType4: Length is 4 byte1002: Data is RM_MONITOR_REQUEST_OBJECT

High Availability Monitors

High availability monitors provide additional logging support.




	NOTE: Logging will occur at every polling interval. This can create a very large `syslog file`, so you may want to only use logging when you are troubleshooting.

Entries in /var/adm/syslog/syslog.log are marked with the monitor daemon name, for example pkgmond or fsmond, followed by the resource name and logging data. Additions, deletions, notifications, and changes in resource states are logged. Errors explaining why a resource is not available for monitoring, or why the monitor cannot access a resource are also logged there.

Look at the registrar.log if you are having trouble finding resources that you suspect exist on your system. This log contains any errors that were encountered when trying to read the dictionary. If a dictionary is corrupt, the registrar will not be able to read it, and EMS will not be able to find the resources associated with that dictionary.

EMS Tracing

Some monitors provide tracing, which can be used for debugging monitor code.

Use the -d option to turn on tracing for EMS. Tracing should only be used at the request of your HP support personnel when trying to determine if there may be a problem with EMS. To turn on tracing, modify the .dict file in /etc/opt/resmon/dictionary and add -d to the monitor you like to trace:

MONITOR: /etc/opt/resmon/lbin/mibmond -l -d

Kill the monitor process. The monitor will automatically restart with tracing enabled. To speed up monitor restart, use the resls command with the top level of the resource class as an argument, for example, resls /system.

Tracing is customarily logged to /var/opt/resmon/log/monitor_name.log. The monitor_name usually matches the name used for the monitor in the dictionary file. For example, the MIBmonitor uses mibmond.dict and mibmond.log.