Managing System Performance

This section provides some guidelines and suggestions for improving the performance of a system or workgroup.

Performance Bottlenecks

A system may perform slowly or sluggishly for a variety of reasons, and you may need to do considerable investigation to determine the source of bottlenecks on a given system. You need to consider the interrelationships between the different components of the system, not just its individual components. Start with the tools described under “Measuring Performance”.

Once you’ve isolated a performance problem and you decide how to address it, change only one thing at a time. If you change more than one thing, you will not know which change helped performance. It’s also possible that one change will improve performance while another makes it worse, but you won’t know that unless you implement them separately and measure performance in between.

The following shows some possible system bottlenecks:

CPU Bottlenecks:

Many background processes running at a high priority consuming a lot of CPU time, or a “runaway” process. If response time is unacceptable, lower the priority of some processes, and kill any unwanted processes.

Memory Bottlenecks:

high deactivations
high paging activity
little or no free memory available
high CPU usage in System mode

Disk Bottlenecks:

high disk activity
high idle CPU time waiting for I/O requests to finish

long disk queues




	NOTE: Put your most frequently accessed information on your fastest disks, and distribute the workload evenly among identical, mounted disks so as to prevent overload on a disk while another is under-utilized. This can often be accomplished by moving swap areas and heavily accessed file systems off the root disk, or by using disk striping, LVM, and/or disk mirroring to spread I/Os over multiple disks. See also “Checking Disk Load with sar and iostat”.

Network Bottlenecks:

Excessive demand on an NFS server.
LAN bandwidth limitations

Guidelines

Performance is a notoriously difficult topic on which to provide definite advice; these guidelines should not be taken as formal recommendations from HP, but merely as the closest the authors could come to distilling a consensus from the observations of the experts they consulted.

Keep NFS servers and their clients on the same LAN segment or subnet. If this is not practical, and you have control over the network hardware, use switches, rather than hubs, bridges and routers, to connect the workgroup.
As far as possible, dedicate a given server to one type of task.
For example, in our sample network (see “A Sample Workgroup / Network”) flserver acts as a file server, exporting directories to the workstations, whereas appserver is running applications.
If the workgroup needed a web server, it would be wise to configure it on a third, high-powered system that was not doing other heavy work.
On file servers, use your fastest disks for the exported file systems, and for swap.
- Distribute the workload evenly across these disks.
  For example, if two teams are doing I/O intensive work, put their files on different disks or volume groups. See “Checking Disk Load with sar and iostat”.
- Distribute the disks evenly among the system’s I/O controllers.
For exported HFS file systems, make sure the NFS read and write buffer size on the client match the block size on the server.
You can set these values when you import the file system onto the NFS client; see the Advanced Options pop-up menu on SAM’s Mounted Remote File Systems screen. See “Checking NFS Server/Client Block Size” for directions for checking and changing the values.
Enable asychronous writes on exported file systems.
See “Checking for Asynchronous Writes”.
Make sure enough nfsd daemons are running on the servers.
As a rule, the number of nfsds running should be twice the number of disk spindles available to NFS clients.
For example, if a server is exporting one file system, and it resides on a volume group comprising three disks, you should probably be running six nfsds on the server.
For more detail, see “Checking for Socket Overflows with netstat -s” and “Increasing the Number of nfsd Daemons”.
Make sure servers have ample memory.
Efforts to optimize disk performance will be wasted if the server has insufficient memory.
Monitor server memory frequently (see “Measuring Memory Usage with vmstat”; and never prepare a hardware budget that doesn’t include additional memory!
Defragment servers’ JFS file systems regularly.
Fragmentation means that files are scattered haphazardly across a disk or disks, the result of growth over time. Multiple disk-head movements are needed to read and update such files, theoretically slowing response time.
In practice, though, a server is dealing with many I/O requests at a time, and intelligence is designed into the drivers to take account of the current head location and direction when deciding on the next seek.
This means that defragmenting an HFS file system on HP-UX may never be necessary; JFS file systems, however, do need to be defragmented regularly.
See “Defragmenting an HFS File System” and “Defragmenting a JFS File System”.
Keep exported files and directories as small as possible.
Large files require more NFS operations than small ones, and large directories take longer to search.
Encourage your users to weed out large, unnecessary files regularly (see “Finding Large Files”).
Monitor server and client performance regularly.
See “Measuring Performance”.

Resource Hogs

To get an idea of your top CPU hogs, run SAM and select Performance Monitors. (On pre-10.20 systems select Process Management, then Performance Monitors.) Then select Processes With Highest CPU Usage. (Or run /usr/bin/top from the command line.)

To compare memory use by the processes currently running, run ps -efl. Look under the SZ column of the resulting display.

Measuring Performance

The saying, “you can’t manage what you don’t measure,” is especially true of system and workgroup performance. Here are some ways to gauge your workgroup’s performance against the “Guidelines” earlier in this section.

Checking Disk Load with sar and iostat

To see how disk activity is distributed across your disks, run sar -d with a time interval and frequency, for example:

sar -d 5 10

This runs sar -d ten times with a five-second sampling interval. The %busy column shows the percentage of time the disk (device) was busy during the sampling interval.

Compare the numbers for each of the disks the exported file systems occupy (note the Average at the end of the report).

Another way to sample disk activity is to run iostat with a time interval, for example:

iostat 5

This will report activity every five seconds. Look at the bps and sps columns for the disks (device) that hold exported file systems. bps shows the number of kilobytes transferred per second during the period; sps shows the number of seeks per second (ignore msps).

If some disks exporting file systems are consistently much busier than others, you should consider redistributing the load. See “Extending a Logical Volume to a Specific Disk ” and “Moving Data to a Different Physical Volume ”. If you decide to move a directory to a different server, the cookbook for “Moving a Directory to a Logical Volume on Another System” may be helpful.




	NOTE: On disks managed by “The Logical Volume Manager (LVM)”, it can be hard to keep track of what file systems reside on what disks. It’s a good idea to create hardcopy diagrams of your servers’ disks; see “Diagramming a System’s Disk Usage”.

Checking NFS Server/Client Block Size

In the case of an HFS file system, the client’s NFS read/write block size should match the block size for that file system on the server.

On the NFS server, you can use dumpfs to check the blocksize for an HFS file system; for example:

dumpfs /work | grep bsize

In the resulting output, bsize is the block size, in bytes, of the file system /work.




	NOTE: For a JFS file system, you can use mkfs -m to see the parameters the file system was created with. But adjusting the client’s read/write buffer size to match is probably not worthwhile because the configured block size does not govern all of the blocks. See “Examining File System Characteristics”.

On the NFS client, use SAM to check read/write block size.
Go to Networking and Communications/Networked File Systems/Mounted Remote File Systems, select each imported file system in turn, pull down the Actions menu and select View More Information, then View Mount Options.
Read Buffer Size and Write Buffer Size should match the file system’s block size on the server.
If it does not, you can use SAM to change it.
NOTE: Unmount the file system on the NFS client first.
Go back to the Mounted Remote File Systems screen, select the file system whose read/write buffer sizes you need to change, pull down the Actions menu and select Modify, then modify the buffer sizes on the Advanced Options screen.

Checking for Asynchronous Writes

Enabling asynchronous writes tells the NFS server to send the client an immediate acknowledgment of a write request, before writing the data to disk. This improves NFS throughput, allowing the client to post a second write request while the server is still writing out the first.

This involves some risk to data integrity, but in most cases the performance improvement is worth the risk.

You can use SAM to see whether asynchronous writes are enabled on a server’s exported file systems.

Run SAM on the NFS server, go to Networking and Communications/Networked File Systems/Exported Local File Systems, select each exported file system in turn, pull down the Actions menu and select View More Information. This screen shows Asynchronous Writes as either Allowed or Not Allowed.

You can change the setting of the Asynchronous Writes flag in SAM, while the file system is still mounted and exported.

Go to Exported Local File Systems, select the exported file system for which you want to allow (or prevent) asynchronous writes, pull down the Actions menu and select Modify. Then select Yes or No under Asynchronous Writes.

Checking for Server Overload with nfsstat -rc

Run nfsstat -rc on an NFS client to get an idea of how the server is performing.

You’ll get a report that looks like this:

Client rpc:
calls      badcalls   retrans    badxid     timeout    wait       newcred
43467543   848        6          3868       27942      0          0

badxid should be small in relation to timeout. If these numbers are nearly the same, it may mean the server is overloaded and generating duplicate replies to RPC requests that have timed out and been retransmitted. Check the server’s memory, disk and NFS configuration; see the “Guidelines” in the previous section.




	NOTE: A `badxid` that is close to zero and a large number for `timeout` may indicate packets are being dropped; that is, the client’s requests are timing out because they never reach the server. In this case the problem is likely to be a network card on the server or client, or the network hardware.

Measuring Memory Usage with vmstat

vmstat displays a wealth of information; use the -n option to make it more readable on an 80-column display.

The column to watch most closely is po. If it is not zero, the system is paging. If the system is paging consistently, you probably need more RAM.

Checking for Socket Overflows with netstat -s

Although many different processes use sockets, and can contribute to socket overflows, regular socket overflows on an NFS server may indicate that you need to run more nfsd processes. The command,

netstat -s | grep overflow

will show you a cumulative number for socket overflows (since the last boot). If you see this number rising significantly, and NFS clients are seeing poor response from this server, try starting more nfsds; see “Increasing the Number of nfsd Daemons”.

Checking for Network Overload with netstat -i

If you have followed all the “Guidelines” and are still seeing poor response time, the problem may be with the network itself - either with a particular piece of hardware or with the configuration of the network.

To see cumulative statistics on a server, run

netstat -i

If your system has been running for a long time, the numbers will be large and may not reliably reflect the present state of things. You can run netstat iteratively; for example

netstat -I lan0 -i 5

In this case (after the first line), netstat reports activity every five seconds.

Input and output errors should be very low in relation to input and output packets - much less than 1%. A higher rate of output errors on only one server may indicate a hardware problem affecting the server’s connection to the network.

Collisions (colls) should be less than 5%; a higher rate indicates heavy network use which your users are probably experiencing as poor performance. Network traffic and configuration may be beyond your control, but you can at least raise a flag with your network administrator.

Making Changes

Increasing the Number of nfsd Daemons

To increase the number of nfsds running on a server, do the following steps:

Edit /etc/rc.config.d/nfsconf, raising the value of NUM_NFSD; for example:
NUM_NFSD=8
Stop and restart the nfs.server script:
/sbin/init.d/nfs.server stop
/sbin/init.d/nfs.server start

Defragmenting an HFS File System

Defragmenting an HFS file system could improve throughput by reducing disk seek time. In practice, though, most experts believe it will usually make little or no difference to performance. You should do it only if you have good reason to believe, or have received expert advice, that your system will really benefit.




	NOTE: This applies only to HFS file systems. JFS file systems do need to be defragmented regularly. See “Defragmenting a JFS File System”.

You can defragment an HFS file system by backing it up to tape, removing and recreating it, then recovering the data from the tape.

The example that follows shows an alternative method, using dcopy, and assumes you have enough disk space to create a new logical volume at least as large as /dev/vg01/lvol8. We’ll operate on the /work file system, which resides on the logical volume /dev/vg01/lvol8.

Back up the file system; for example,
tar cv /work
backs up /work to the system default tape device, /dev/rmt/0m.
Create a new logical volume (see “Adding a Logical Volume”) but do not mount it to any file system.
We’ll assume this new logical volume is /dev/vg01/lvol9.
Make sure no one has files open in /work and that it is no one’s current working directory, for example:
fuser -cu /work
Unmount /work:
umount /work
Write out the contents of /work to /dev/vg01/lvol9:
dcopy -v /dev/vg01/rlvol8 /dev/vg01/lvol9
NOTE: The source file system should be a raw device (/dev/vg01/rlvol8) and the destination file system should be a block device (/dev/vg01/lvol9).
Mount the new logical volume to the mount point of the original file system, /work:
mount /dev/vg01/lvol9 /work
You can now reuse the original logical volume /dev/vg01/lvol8) or remove it (see “Removing a Logical Volume”).

Configurable Kernel Parameters

In some cases, you may be able to get the results you need by resetting kernel parameters. For example, if a user frequently runs out of processes (symptom no more processes), raising the value of maxuprc might be the answer.




	NOTE: Tunable kernel parameters can be static or dynamic (not requiring a system reboot or kernel rebuild). The list of dynamic tunables is continually growing. To determine which tunables are dynamic on your HP-UX 11i system, use the kmtune command (see the kmtune(1M) manpage), or see the Kernel Configuration portion of SAM. In SAM’s Configurable Parameters screen, administrators can tell at a glance whether or not the value of a particular tunable can be changed without a reboot. As of HP-UX 11i v2, use the kctune command or the kcweb web interface. See kctune(1M) and kcweb(1M).

SAM allows you to view and change kernel parameter settings. To view or adjust parameters, select Kernel Configuration and then Configurable Parameters. Then select Help/Overview, scroll down to the link for Configurable Kernel Parameters and select it; then scroll down till you find the parameter you are interested in and select it. Another way to get help on a single parameter is to select that parameter on the Configurable Parameters screen, then press the F1 function key.

For more information on dynamic tunables, see “Reconfiguring the Kernel
(Prior to HP-UX 11i Version 2)” and the Dynamically Tunable Kernel Parameters in HP-UX 11i whitepaper at http://docs.hp.com.




	CAUTION: Make sure you read the help for all the parameters related to any parameter you are considering changing. In the case of `maxuprc`, you would need to read the help on `nproc` as well as `maxuprc`.

Other Performance Management Tools

Some of the tools that HP provides are:

HP also provides several sources for tools and support for HP-UX. See http://www.software.hp.com. This web page has links to:

HP-UX 3rd party and public domain software
This catalog contains over 1000 packages in binary and source format. Each package is placed into a single category on the archive. These categories can be viewed in alphabetical or chronological order.
HP-UX application demos, shareware, and freeware
HP patches
On-demand support

SAM

The System Administration Manager (SAM) tool allows you to perform many system administration tasks without having to know all the HP-UX commands involved. In fact, SAM provides a good means of learning the HP-UX commands needed for a given task - it records its actions, including the HP-UX commands it has used, in a log, which you can look at by pulling down the Options menu on any SAM screen.

For more information on SAM’s capabilities, use SAM’s online help or see the manpage sam(1M). See also “Using System Administration Manager (SAM) ”.

To start SAM, enter:

/usr/sbin/sam

The top Command

Use the top command to see processes ranked by CPU usage. See the manpage top(1).

To run top, enter:

/usr/bin/top

OpenView Products

A broad portfolio of OpenView based products to help you manage your HP-UX and Windows NT based systems is available from HP and HP OpenView Solutions Partners. HP OpenView products are available to help you:

Manage logins on HP-UX systems (and other operating systems)
Monitor the performance of HP-UX systems
Manage databases
Manage electronic mail (e-mail)
Manage Simple Network Message Protocol (SNMP) applications and resources

and a lot more. Some of the products are:

“GlancePlus and GlancePlus Pak”
IT/Administration
IT/Operations
MeasureWare
Storage Management
Openspool
PerfView
Software Distributor
VantagePoint
Network Management
Security Management

For complete and current information on HP OpenView products, service, and support, go to http://www.openview.hp.com.

GlancePlus and GlancePlus Pak

HP GlancePlus is a diagnostic performance tool which provides detailed immediate performance information about your system. It has built-in bottleneck alarms and zoom-in capabilities to make performance troubleshooting easier.

The HP GlancePlus Pak combines the HP GlancePlus and HP MeasureWare products. This provides both detailed immediate diagnostic and long-term analysis for performance data. These software products are available on multivendor platforms as well as for HP-UX.

HP MeasureWare Agent is a comprehensive long-term performance tool which collects, alarms on, and manages system performance information as well as metrics from other sources such as database probes. It provides data and alarms for PerfView, HP OpenView NNM or IT/Operations as well as third-party products.

Kernel Resource Monitor (KRM)

The Kernel Resource Monitor is included with Event Monitoring Systems (EMS) Hardware Monitors. The KRM checks HP-UX resources such as nproc (number of processes) which are controlled by the kernel parameters. KRM continually checks the actual usage of these resources. If the amount of the usage meets or exceeds a preset value, you are notified by e-mail, console message, system log, or other means.

This can be useful for tuning the kernel parameters for your particular system and avoiding panics and performance problems caused when usage of HP-UX resources approaches too high a level.

The EMS Monitors can be integrated with applications responsible for maintaining system availability, such as MC/ServiceGuard. If configured to do so, they can provide event notification to system management applications such as HP OpenView IT/Operations and HP Network Node Manager.

The EMS Hardware Monitors use the same EMS framework as the EMS High Availability (HA) monitors. The HA EMS monitors are a separate set of monitors available at additional cost.

Some of the hardware monitors for fibre channel products write event information to text logs read by a new Predictive scanner, emsscan, which in turn may send events to the Response Center via On-line Predictive.

The EMS Hardware Monitors (including the Kernel Resource Monitor) are distributed on the Support Plus CD media and available to download from http://software.hp.com.

Select “Enhancement Releases” and then “Support Tools for the HP 9000.”

For more information see Support Plus: Diagnostics User’s Guide, and EMS Hardware Monitors User’s Guide on the Instant Information CD or at http://docs.hp.com/hpux/systems/.