The Logtool Utility allows you to monitor recoverable errors detected by the computer, such as single-bit memory errors and I/O device errors. By analyzing this information, you can troubleshoot failures before they become acute. (For a quick start, see Viewing A Log.)
Error information is contained in system log files. Since the raw files are hard for humans to read, Logtool allows you to format the files and to filter them for desired entries (for example, for all entries related to SCSI disks).
Unlike other support tools, this utility program is not device specific and does not require selection of a device to run. A license is not required to run the Logtool Utility.
File Commands
SelectRaw (SR) - Select a raw log file(s) to view, filter or format FormatRaw (FR) - Format the currently selected raw log file SelectFormatted (SF) - Select a formatted log file(s) to view or filter SwitchLog (SL) - Create a new "current" raw log file Save (SAV) - Save Logtool's main window Print (PRN) - Print Logtool's main window Exit (EX) - Exit Logtool and return to the STM prompt
View Commands
RawSummary (RS) - View the summary of the currently selected raw log file(s) RawLog (RL) - View the currently selected raw log file(s) FormattedSummary (FS) - View the summary of the currently selected formatted log file(s) FormattedLog (FL) - View the currently selected formatted log file(s)
Filters Commands
Format (FF) - Select format filters for the currently selected raw log file(s) DisplayRaw (DR) - Select display filters for currently selected raw log file(s) DisplayFormatted (DF) - Display the currently selected formatted log file(s)
Memory Commands
ClearLog (CL) - Clear the memory log file SetTimer (ST) - Change the polling interval for logging memory errors ViewDealloc (VDA) - View errors pertaining to deallocated memory errors ViewDetail (VD) - View the entire memory log file
Chassis Commands
View Boot Log (CB) - View the Chassis Code Boot Log file View Error Log (CE) - View the Chassis Code Error Log file
Help Commands
Help (HE) - Display online help for Logtool Version (VERS) - Display Logtool version and copyright information
Control-C - Interrupt the tool.
The Logging Daemon
When STM is installed on a computer, a logging daemon is activated to monitor recoverable errors that have been detected by the system, for example:
Logging
When the logging daemon detects an event, it logs an entry in a system log file located (by default) in the directory "/var/stm/logs/os". The raw log file names have the format "logN.raw," where N is a number that starts at 1 and is incremented each time a new log file is created with the command SwitchLog (SR). The only exception to the raw log file naming convention is the log file to which the logging daemon is currently logging data; this file is called "logN.raw.cur."
Memory errors, when detected by the memory logging daemon, are logged into the memory log file called memlog.
The chassis code logging daemon logs information to the chassis code error log file and boot log files called ccerrlog and ccbootlog respectively. These log records are generated by various system hardware, software and firmware modules. They can contain error indications as well as informational messages. Chassis code logging was implemented beginning with the introduction of N-class systems on HP-UX 11.00, and will be featured on some newer systems as they are released.
Logging Control
To change the file to which the daemon is logging data, enter the SwitchLog (SL) command.
You can modify the behavior of the logging daemon by editing the configuration file "diaglogd.cfg" located in the directory "/var/stm/config/sys". The file itself contains documentation regarding its format.
Logtool and the Logging Process
The role of Logtool in the logging process is shown in the diagram and is explained in more detail below.
errors
|
v logger
STM logger <- config
| file
v +-------------+
FILE: log1.raw -> | Raw Display | -> DISPLAY
| | Filter | of raw
v +-------------+ data
+---------------+
| Format Filter |
+---------------+
|
v +-------------------+
FILE: log1.fmt1 -> | Formatted Display | -> DISPLAY of
| Filter | formatted
+-------------------+ data
Raw Log Files
The first step in analyzing the log files is usually to enter the SelectRaw (SR) command at the Logtool Utility> prompt. A summary of the contents of the log file(s) will automatically be displayed. In addition the summary information can be obtained independently using the RawSummary (RS) command. To view the entries in the log file, you use the RawLog (RL) command.
Looking at all the entries in the raw log file can be burdensome:
To solve these dilemmas, Logtool offers the ability to format and filter the entries in the log files.
Formatting the Data
To translate the raw log file into a more readable format, enter the FormatRaw (FR) command at the Logtool Utility> prompt. The formatted data is put into a file with the name "logN.fmtN" where the first number (N) matches that of the raw log file and the second number (N) starts at 1 and is incremented for each new formatted file generated from the same raw log file. If multiple files have been chosen, the file name will have a plus sign ("+") in it. For example, log1.raw and log2.raw will be formatted into log1+.fmt1.
To choose a particular formatted log file(s) for viewing or filtering, use the SelectFormatted (SF) command. A summary of the contents of the formatted log file(s) will automatically be displayed. In addition the summary information can be obtained independently with the FormattedSummary (FS) command. To view the entries in the formatted log file, you use the FormattedLog (FL) command.
Filtering the Data
Even if the data is formatted, there is generally a large number of entries contained in a typical log. When you are troubleshooting a problem, only certain entries may be relevant. For example, you may want to see only I/O errors that occurred during a certain time period on a certain hardware path.
Logtool offers three different filters to select the entries relevant to your task.
Memory Log
The memory log is handled somewhat differently than the system logs described above. Commands for memory log options are:
Chassis Logs
The commands to view the Chassis error log file or Chassis boot log files are found in the Chassis Menu. The user can select which file they want to view from the displayed list of the current files. The commands for chassis logs are:
The chassis error logs are created by the logging daemon (diaglogd). Chassis Boot Log entries can come from various system hardware, software and firmware modules. The Chassis Error log contains entries for chassis codes reported with an alert level indicating that an error has occurred. For instance, chassis codes are logged if problems occur with the power supplies. By examining the chassis error log, you could see a record of these power supply problems.
The entry types currently logged by Logtool include:
Logged by I/O system drivers, I/O error entries typically contain information concerning exception conditions that the driver encountered.
Logged by the system when a Low Priority Machine Check (LPMC) condition has occurred. An LPMC is a recoverable error detected by hardware; the hardware is able to recover from the error and continue operation, though perhaps in a degraded fashion. In contrast, a High Priority Machine Check is an error detected by hardware from which it cannot recover.
Logged by the system when hardware detects a recoverable over-temperature condition (over-temp warning).
Logged by the Logical Volume Manager, typically containing information concerning exception conditions the manager encountered.
Logged by the logger process. The type of events that are logged include suspension and resumption of logging.
To run the Logtool Utility:
Logtool offers many different ways to select log entries and view them. A typical method for viewing logs involves these steps:
For example, you may have used a format filter that resulted in all entries from the SCSI disk driver being formatted. After viewing a summary of the formatted entries, you may decide that only entries for a particular device are of interest or, perhaps, only entries after a certain date and time. You could then set the formatted display filter to specify a specific device path or date and time and re-display the formatted log file.
To troubleshoot a disk problem (sample procedure):
Follow the general procedure described in Viewing A Log, but specify disk-related parameters in any of the filters commands.
For example, de-activate the selections for all of the entry types except I/O Errors, such as "Low Priority Machine Check", "LVM Entries", etc., and activate the selection "Filter I/O Errors" under "I/O Error Entries". Under "Products:", select only the relevant products (for example select "SCSI Disk" and not "LAN Interface").
The method for viewing the memory error log file involves these steps:
If most of the errors in the file indicate the same slot for the memory controller, then the SIMM module in that particular slot is probably bad.
If occasional errors report the same slot for the memory controller, then the SIMM module in that particular slot is suspect.
If the same bit number is indicated across many slots then the memory controller is bad.
Consult the tool's Activity Log, accessible from the "cstm>" prompt.
The Logtool Utility may be waiting for a response to a query message it is displaying. Answer the query message and continue.
Entries in the raw system log are hard to understand; format the log to make the entries more readable (FormatRaw (FR) command).
Specify that only certain entries in the log file be formatted or displayed using either: Format (FF), DisplayRaw (DR) or DisplayFormatted (DF).
Start a new current system log (SwitchLog (SL) command).
At present, Logtool only supports the formatting of log files that originated on the host system. Format the log files on the computer on which they were generated. Once a log has been formatted, it can be moved to other computers for display and filtering.
The Logtool Utility offers the following commands:
SelectRaw (SR)
The "Select Raw" or "SR" command allows you to select one or more raw log files for viewing, filtering or formatting. If multiple files are selected, all of the entries are combined together and treated as one file. Once the file is selected, the summary information is automatically displayed.
The current raw log file has the extension ".cur". Each time a new log file is generated, its file name is incremented by one. For example, the initial file will be "log1.raw.cur". When a new file is generated, "log1.raw.cur" will become log1.raw, and the current raw log file will be "log2.raw.cur".
If the current raw log file is selected, the output of subsequent commands such as RawLog (RL) or FormatRaw (FR), may reflect additional records/and or counts that weren't displayed initially. This is because more errors may have been logged to the currently active file since it was last read.
NOTE: Users can optionally specify a file of their own choosing but are cautioned that only log files generated on the host system should be formatted on that system. Attempts to format log files from other systems may result in formatting problems or incorrect formatted output. Once a formatted log file is generated, it can be moved to other systems and displayed without any problems.
Format Raw (FR)
The "Format Raw" or "FR" command formats entries from the currently selected raw log file or files chosen via the SelectRaw (SR) command. If a file has not been selected, the current raw log file, if there is one, is used as the default. Formatting makes the log entries much easier for humans to understand.
NOTE: Only log files generated on the host system should be formatted on that system. Attempts to format log files from other systems may result in formatting problems or incorrect formatted output. Once a formatted log file is generated, it can be moved to other systems and displayed without any problems.
The entries in the raw log file(s) are filtered according to the Format filter settings (Format (FF) command). Entries which do not match the filter are not formatted. If you wish to format entries which were previously filtered out, change the filter and reformat the log file. If no filters have been selected all entries will be formatted.
The data portion of each entry is formatted into text by the appropriate log decoding routine. If an appropriate log decoding program cannot be located for a particular entry, the data portion of the entry is displayed in hex.
Formatted log file names have the form "logN.fmtN" where the first number (N) matches that of the raw log file from which it was generated and the second number (N) starts at 1 and is incremented for each new formatted file that is generated from the same raw log file. For example, "log1.raw" becomes "log1.fmt1" the first time it is formatted. If you then format the same raw log file a second time, the newly formatted file is named "log1.fmt2".
After a raw file is formatted, the formatted log summary is automatically displayed. The newly created formatted log file becomes the selected formatted file to be displayed or summarized by the FormattedSummary (FS) and FormattedLog (FL) commands.
SelectFormatted (SF)
The "Select Formatted" or "SF" command allows you to select one or more formatted log files for viewing or filtering. Once the file(s) is selected, the summary information is automatically displayed.
SwitchLog (SL)
The "Switch Log" or "SL" command moves the existing current raw log file "logN.raw.cur" to "logN.raw." The next time the logger process needs to write an error it opens a new "logN.raw.cur" file, incrementing the "N" by one.
For example, the Switch Log command moves "log5.raw.cur" to "log5.raw," causing the logger process to open a new raw log file called "log6.raw.cur."
Individual log files can be kept from becoming too large by issuing this command occasionally. Alternatively, a size threshold can be set in the diaglogd.cfg file.
Save (SAV)
The "Save" or "SAV" command saves the contents of Logtool's main window. (Only the most recent 200 lines can be saved.)
Print (PRN)
The "Print" or "PRN" command prints the contents of Logtool's main window. (Only the most recent 200 lines can be printed.)
Exit (EX)
The "Exit" or "EX" command is used to leave the Logtool Utility and return to the "cstm>" prompt.
RawSummary (RS)
The "Raw Summary" or "RS" command can be selected independently but is also run automatically when file(s) are selected. It displays a summary of the entries in the currently selected raw log file(s). If no files were selected, the current raw log file, if there is one, is used as the default.
The summary reports the total number of each type of error entry that occurs in the file(s) (for example, "Number of LPMC entries: 6"). If multiple files are selected the entries are combined together and treated as one file.
The summary reports additional information on I/O error entries, listing the specific device paths, products, product qualifiers, and logger events that occur in the file(s), and the number of times they occur.
For more information, see Log Entry Types.
RawLog (RL)
The "Raw Log" or "RL" command displays the entries from the currently selected raw log file(s). The header portion is displayed in text and the data portion is displayed in hex.
The raw log file(s) could have been selected via the SelectRaw (SR) command. Filters may have been selected and applied via the DisplayRaw (DR) command. If a raw log file was not selected, then the current raw log file, if there is one, is used as the default. Entries which do not match the filter are not displayed. If you wish to display entries which were previously filtered out, change the filter and redisplay the log file using the DisplayRaw (DR) command. If multiple files are selected the entries are combined together and treated as one file.
The top of the display in the "Display Filters: area" lists the filter criteria specified at the time the raw log file was filtered.
FormattedSummary (FS)
The "Formatted Summary" or "FS" command can be selected independently but is also run automatically when file(s) are selected. It displays a summary of the entries in the currently selected formatted log file(s). If no files were selected via the SelectFormatted (SF) command then the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file. If multiple files have been selected the entries are combined together and treated as one file.
The summary reports the number of each type of error entry that occurs in the file(s) (for example, "Number of LPMC entries: 6").
The summary reports additional information on I/O error entries, listing the specific device paths, products, product qualifiers, loggers, device types and device qualifiers that occur in the file(s), and the number of times they occur.
For more information, see Log Entry Types.
FormattedLog (FL)
The "Formatted Log" or "FL" command displays the formatted log data that was generated by the FormatRaw (FR) command.
The formatted log file(s) could have been selected via the SelectFormatted (SF) command from the File menu. If a file was not selected, the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file. Filters may have been selected and applied via the DisplayFormatted (DF) command. Entries which do not match the filter are not displayed. If you wish to display entries which were previously filtered out, change the filter and redisplay the log file.
If multiple files have been selected the entries are combined together and treated as one file.
The top of the display in the "Display Filters: area" lists the filter criteria specified at the time the raw log file was filtered.
Format (FF)
The "Format" command allows you to select filtering criteria to be applied to the entries in the raw log file(s) before formatting.
In order to set the filter, you should have first selected a raw log file(s) via the SelectRaw (SR) command. If a file was not selected, then the current raw log file, if there is one, is used as the default.
Items which occur in the raw log file(s) are displayed as candidates for filtering. These items are displayed on the screen and you can select as many items as desired.
Once one or more items have been selected, subsequent FormatRaw (FR) commands (under the File menu) only format entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display Entries of the following types" has an item beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by answering "no" to the prompt at the appropriate entry type that you're not interested in seeing.
The I/O Error Entry types have a separate section which is controlled by radio buttons and are presented as numbered choices. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries, enter the number that corresponds to the "No I/O Errors". If you wish to filter on a specific qualifier you can enter the number that corresponds to the label "Filter I/O Errors" and then make your selections from the choices presented below by selecting the appropriate number (or numbers) and entering them at the prompt. Separate the items with spaces or commas.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
DisplayRaw (DR)
The "DisplayRaw" or "DR" command is used to select which entries in the raw log file(s) to display.
In order to set the filter, you should have first selected a raw log file(s) via the SelectRaw (SR) command. Otherwise, the current raw log file, if there is one, is used as the default. Items which occur in the raw log file(s) are displayed as candidates for filtering. These items are displayed on the screen and you can select as many items as desired.
Once one or more items have been selected, subsequent Raw Log commands only display entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display Entries of the following types" has an item beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by answering "no" to the prompt at the appropriate entry type that you're not interested in seeing.
The I/O Error Entry types have a separate section which is controlled by radio buttons and are presented as numbered choices. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries, enter the number that corresponds to the "No I/O Errors". If you wish to filter on a specific qualifier you can enter the number that corresponds to the label "Filter I/O Errors" and then make your selections from the choices presented below by selecting the appropriate number (or numbers) and entering them at the prompt. Separate the items with spaces or commas.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
In addition to specifying criteria for each of the header fields, you can enter a hex pattern to search for in the data for each entry. If a search pattern is specified, it is applied in addition to any other fields already specified.
NOTE: Specifying a search pattern may result in a noticeable delay because each entry's data must be searched. When a search pattern is not used, only header contents are examined.
DisplayFormatted (DF)
The "DisplayFormatted" or "DF" command is used to select which entries in the formatted log file(s) to display.
In order to set the filter, you should first select one or more formatted log files via the Select Formatted (SF) command. If a file was not selected, then the current raw log file, if there is one, is automatically formatted and becomes the selected formatted file.
Items which occur in the log file(s) are displayed as candidates for filtering. These items are displayed on the screen and you can select as many items as desired.
Once one or more items have been selected, subsequent Formatted Log commands (under the View menu) only display entries which match the filter criteria. An entry is considered to be a match for a particular filter field if it matches any of the selected items that were set in that field. If a field has no items selected, any entry matches that field. In order for an entry to pass through the filter, it must match the criteria for all of the filter fields.
For example, if you select a Device Type of "Disk" and a Device Qualifier of "Hard," only entries for Hard disks are formatted/displayed. If you select a Device Path of "1/0/0" and a Device Type of "Disk," no entries are formatted/displayed unless a disk drive is associated with that path.
The line labeled "Display Entries of the following types" has an item beneath it for each of the entry types except I/O entries. These start out as selected items and may be unselected by answering "no" to the prompt at the appropriate entry type that you're not interested in seeing.
The I/O Error Entry types have a separate section which is controlled by radio buttons and are presented as numbered choices. The default selection is to show all I/O error entries. If you don't want to see any I/O error entries, enter the number that corresponds to the "No I/O Errors". If you wish to filter on a specific qualifier you can enter the number that corresponds to the label "Filter I/O Errors" and then make your selections from the choices presented below by selecting the appropriate number (or numbers) and entering them at the prompt. Separate the items with spaces or commas.
The starting and ending date/time fields initially contain the date/time of the first and last log entry.
ClearLog (CL)
The "Clear Log" or "CL" command notifies the memory logging daemon to create a new memory log file.
SetTimer (ST)
The "Set Timer" or "ST" command allows you to change the interval at which the memory logging daemon polls the memory for errors.
ViewDealloc (VDA)
The "View Dealloc" or "VDA" command allows you to display the deallocated page information from the memory log file.
ViewDetail (VD)
The "View Detail" or "VD" command allows you to display the entire contents of the memory log file.
Help (HE)
The "Help" or "HE" command displays online help for the Logtool Utility (the help system currently being viewed).
For online help on STM as a whole, use the Help menu in the STM main window.
Version (VERS)
The "Version" or "VERS" command displays version and copyright information for the Logtool Utility.
Control-C
Pressing the "Control-C" keys allows you to interrupt the tool. When "Control-C" is detected, a list of choices will be displayed on the screen. The list includes "Abort", "Interrupt", "Escape" and "Continue".
th "Abort" and "Escape" will cause the display of the "cstm>" prompt. "Abort" will cause the tool to be aborted. When the tool actually exits, the state of the tool execution in the system map is set to ABORTED.
"Escape" will not terminate the execution of the tool. It allows you to disconnect from the currently executing tool and return to the "cstm>" prompt. To reconnect to the tool, use the Attach command in the "cstm>" prompt.
"Continue" causes the tool to continue with what it was doing at the time of Control-C.
"Interrupt" sends an interrupt to the tool. When the tool receives the interrupt, it stops whatever it is doing and prompts you if you want to continue with the operation. If the reply is "Continue", the tool resumes what it was doing at the time of the interrupt. If the reply is "Abort", it terminates the operation and returns to the "Logtool Utility>" prompt.
The tool may not immediately receive the interrupt as it could be in the middle of an operation that should not or could not be interrupted. The interrupt is received as soon as the tool completes the operation.
The tool will receive and ignore the interrupt command when it is already in the process of returning to the "Logtool Utility>" prompt.
The shortcut commands described below were implemented beginning with version 2.0 of Logtool.
When entering Logtool, there are several "shortcut" commands that can be invoked. The shortcut commands automatically select the current raw log file (*raw.cur) as the basis for all of their operations. If there is not a current raw log file to use, the user is informed that they need to select a file.
In addition, the SelectRaw (SR) and SelectFormatted (SF) commands automatically display the file's summary information once a file has been selected.
Once a file has been selected, the raw log summary is automatically displayed.
Defaults to the current raw log file, if there is one, and displays the raw log summary.
Defaults to the current raw log file, if there is one, and displays the raw log file. The header portion of the file entries is displayed in text; the data portion is displayed in hex.
Defaults to the current raw log file, if there is one, and presents the dialog to the user.
Defaults to the current raw log file, if there is one, and presents the dialog to the user.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log summary.
Once a file has been selected, the formatted log summary is automatically displayed.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log summary.
Defaults to the current raw log file, if there is one, automatically formats the file, and displays the formatted log file.
Defaults to the current raw log file, if there is one, automatically formats the file, and presents the dialog to the user.
Throughout this document there are references to multiple file selection capabilities, and the behaviors when multiple files have been selected.
The handling of multiple files was implemented beginning with version 3.0 of Logtool.
See the specific commands for information regarding multiple file handling. In general, multiple files may be selected in SelectRaw (SR) or SelectFormatted (SF). Once selected, all entries from all the selected files are combined together and treated as one file for viewing, filtering and formatting.