Common Problems with NFS

Issue the /usr/sbin/ping(1M) command on the NFS client to make sure the NFS server is up and is reachable on the network. If the ping command fails, either the server is down, or the network has a problem. If the server is down, reboot it, or wait for it to come back up. For information on troubleshooting network problems, see Installing and Administering LAN/9000 Software.

Issue the following command on the NFS client to make sure the server is running all the NFS server processes:

/usr/bin/rpcinfo -p servername

The rpcinfo command should display the following processes:

If any of these processes is not running, follow these steps:

Issue the following command on the NFS client to make sure the rpc.mountd process on the NFS server is available and responding to RPC requests:

/usr/bin/rpcinfo -u servername mountd

If the rpcinfo command returns RPC_TIMED_OUT, the rpc.mountd process may be hung. Issue the following commands on the NFS server to restart rpc.mountd (PID is the process ID returned by the ps command):

/usr/bin/ps -ef | /usr/bin/grep mountd
/usr/bin/kill PID /usr/sbin/rpc.mountd

You can receive "server not responding" messages when the server or network is heavily loaded and the RPC requests are timing out. Try doubling the timeo mount option for the directory, as in the following example from the /etc/fstab file, which changes the timeo value from 7 (the default) to 14. (The timeo option is in tenths of a second.)

cabbage:/usr /usr nfs nosuid,timeo=14 0 0

Issue the following command on the NFS client to check that your hosts database returns the correct address for the NFS server:

/usr/bin/nslookup server_name

If your client cannot resolve the server's hostname, see “If You Receive an "Unknown Host" or "Not In Hosts Database" Message”.

Issue the same nslookup command on the NFS server, and compare the address with the one returned by the nslookup command on the NFS client. If they are different, correct your NIS, BIND, or /etc/hosts configuration. For information on NIS troubleshooting, see “Common Problems with NIS”. For information on BIND or /etc/hosts, see Installing and Administering Internet Services.

If you are using AutoFS, issue the ps -ef command to make sure the automountd process is running on your NFS client. If it is not, follow these steps:

If the "server not responding" message was followed by RPC_AUTH_ERROR; why=AUTH_BOGUS_CREDENTIAL, this could mean that you (or the user who received the message) are a member of too many groups. On HP-UX release 9.0 or later, you can be a member of up to 16 groups. On HP-UX releases prior to 9.0, you can be a member of up to 8 groups.

Issue the following command on the NFS client to check that the NFS server is exporting the directory you want to mount:

/usr/sbin/showmount -e server_name

If the server is not exporting the directory, edit the /etc/exports file on the server so that it allows your NFS client access to the directory. Then, issue the following command to force the server to read its /etc/exports file.

/usr/sbin/exportfs -a

If the directory is exported with the access option, make sure your NFS client is included in the access list, either individually or as a member of a netgroup.

If your NFS client is included in the access list as a member of a netgroup, make sure it is a member of the netgroup in the server's /etc/netgroup file.

If you are using NIS to manage your netgroups, issue the following command to determine whether your NIS server has up-to-date information about the netgroup that includes your client:

/usr/bin/ypmatch netgroup_name netgroup

If your NIS server does not return the correct information, see “Common Problems with NIS”.

Issue the following commands on the NFS server to make sure your NFS client is listed in its hosts database:

nslookup client_name
nslookup client_IP_address

If the server cannot resolve your client's hostname, see “If You Receive an "Unknown Host" or "Not In Hosts Database" Message”.

If rpc.mountd is configured in /etc/inetd.conf on the NFS server, check the server's /var/adm/inetd.sec file to make sure your NFS client is allowed access to rpc.mountd.

Check the mount options in the /etc/fstab file on the NFS client. A directory you are attempting to write to may have been mounted read-only.

Issue the ls -l command to check the HP-UX permissions on the server directory and on the client directory that is the mount point. You may not be allowed access to the directory.

Issue the following command on the NFS server:

/usr/sbin/exportfs

Or, issue the following command on the NFS client:

/usr/sbin/showmount -e server_name

Check the export permissions on the exported directory. The directory may have been exported read-only to your client. The system administrator of the NFS server can use the remount mount option to mount the directory read/write without unmounting it. See “To Change the Default Mount Options”.

If you are logged in as root to the NFS client, check the export permissions to determine whether root access to the directory is granted to your NFS client.

If you are logged in as root to the NFS client, and your client is not allowed root access to the exported directory, check the passwd database on the NFS server to determine whether it contains an entry for user nobody. Without root access, the root user on an NFS client is given the access permissions of user nobody. Also, check whether anonymous users are denied access to the directory (with the anon=65535 export option).

If your client is not allowed root access or anonymous user ID access to the exported directory, log in as a non-root user to get access to the directory.

If you are not running NIS, or if the server is in a different domain from the client, check the passwd databases on the server and the client to make sure you have a valid login on both machines and that your user ID is the same on both machines. If your user ID is unrecognized on the NFS server, you will be granted the permissions of user nobody.

If you were attempting to run a program when you received the "permission denied" message, issue the ls -l command on the NFS server to check whether the program you tried to run has the setuid bit set. If it does, check /etc/fstab to determine whether the directory was mounted with the nosuid mount option. If necessary, remove the nosuid option from the /etc/fstab file, then unmount and remount the directory.

Issue the following commands to trace a lookup of the unknown host:

nslookup
> set swtrace
> hostname

The trace will indicate which name services (BIND, NIS, or /etc/hosts) were queried and in what order. If your host is not performing lookups the way you want, see Chapter 5 “Configuring the Name Service Switch” for instructions on configuring the Name Service Switch.

Type exit to exit from nslookup.

If your host is using the /etc/hosts file to resolve hostnames, edit the file to add or correct the entry for the unknown host. Type man 4 hosts for the correct syntax.

If your host is using NIS to resolve hostnames, see “Common Problems with NIS”.

If your host is using BIND (DNS) to resolve hostnames, see Installing and Administering Internet Services for instructions on troubleshooting BIND.

If you received the "device busy" message while attempting to mount a directory, try to access the mounted directory. If you can access it, then it is already mounted.

If you received the "device busy" message while attempting to unmount a directory, a user or process is currently using the directory. Wait until the process completes, or follow these steps:

If the stale file handle occurred because someone removed a file or directory that was in use, or because a server stopped exporting a directory that was in use, follow these steps:

If someone has restored the server's file systems from backup or issued the fsirand command on the server, follow these steps on each of the NFS clients to prevent stale file handles by restarting NFS:

Check whether the NFS server is up and operating correctly. See “If You Receive an NFS "Server Not Responding" Message”.

If the server is down, wait until it comes back up, or, if the directory was mounted with the intr mount option (the default), you can interrupt the NFS mount, usually with CTRL-C.

If the program uses file locking, issue the following commands (on either the client or the server) to make sure rpc.statd and rpc.lockd are available and responding to RPC requests:

/usr/bin/rpcinfo -u servername status
/usr/bin/rpcinfo -u servername llockmgr
/usr/bin/rpcinfo -u servername nlockmgr
/usr/bin/rpcinfo -u clientname status
/usr/bin/rpcinfo -u clientname llockmgr
/usr/bin/rpcinfo -u clientname nlockmgr

If any of these commands returns RPC_TIMED_OUT, the rpc.statd or rpc.lockd process may be hung. Follow these steps to restart rpc.statd and rpc.lockd:

Make sure the directory is exported from the server with the noasync option (the default). If the directory is exported with the async option, the NFS server will acknowledge NFS writes before actually writing data to disk. Changing an exported directory from async to noasync degrades write performance for that directory.

If users or applications will be writing to the NFS-mounted directory, make sure it is mounted with the hard option (the default), rather than the soft option.

If you have a small number of NFS applications that require absolute data integrity, add the O_SYNC flag to the open() calls in your applications. When you open a file with the O_SYNC flag, a write() call will not return until the write request has been sent to the NFS server and acknowledged. The O_SYNC flag degrades write performance for applications that use it.

If you have a large number of NFS applications requiring absolute data integrity, or if your entire installation needs a high degree of data integrity, set the NUM_NFSIOD variable to 0 in the /etc/rc.config.d/nfsconf file on each client, as follows,

NUM_NFSIOD=0

and issue the following commands to kill all the biod processes (PID is a process ID returned by the ps command):

/usr/bin/ps -ef | /usr/bin/grep biod
/usr/bin/kill PID PID ...

The biod daemons improve performance by handling NFS read and write requests from users and applications. After a write request is passed to a biod daemon, control is returned to the user or application. Running a client without biod daemons degrades NFS performance for all users and applications on that client.

If multiple NFS users will be writing to the same file, add the lockf() call to your applications to lock the file so that only one user may modify it at a time.

If multiple users on different NFS clients will be writing to the file, you must also turn off attribute caching on those clients by mounting the file with the noac mount option. Turning off attribute caching degrades NFS performance.

Issue the following command to check your server's memory utilization:

netstat -m

If the number of requests for memory denied is high, your server does not have enough memory. Consider adding more memory or using a different host as the NFS server.

Issue the ps -ef command on the NFS server, and check for many instances of the same application. Sometimes an application clones itself indefinitely until it uses up all the available inodes on a system.

The default maximum number of inodes shipped with HP-UX tends to be too small for sites that make extensive use of NFS. Follow this procedure to increase the maximum number of inodes on your NFS server: