Troubleshooting the BIND Name Server

This section describes how to identify and correct problems with the BIND name server. It discusses the following topics:




	NOTE: After you configure the BIND name service on your network, the following failures may occur: `rcp` and `remsh` may fail with permission denied messages. `rlogin` may prompt you for a password. These problems are the result of switching to domain names. To correct these problems, you must update other network files. See “Updating Network-Related Files”. If you want to run both BIND and HP VUE, you must have an `/etc/resolv.conf` file on your system, or HP VUE does not start. See “Configuring the Resolver to Query a Remote Name Server”. After you configure the BIND name service, `sendmail` uses the name server’s mail exchanger (`MX`) records for mail routing. See the HP-UX Mailing Services Administrator’s Guide at the URL `http://www.docs.hp.com /hpux/netcom/index.html#Internet%20Services` for information on Sendmail.

Troubleshooting Tools and Techniques

This section describes the available tools for troubleshooting the BIND name server.

The ping Command

Use the ping command to test whether you can look up a specific host name. You can also use it to check network connectivity to the name server.

$ /usr/sbin/ping hostname

If the host name lookup fails, use ping with an IP address to test network connectivity.

$ /usr/sbin/ping IP_address

The nsquery Command

Issue the nsquery command to perform a hosts, passwd, or group lookup, as follows:

/usr/contrib/bin/nsquery lookup_typelookup_query

The nsquery command displays the Name Service Switch configuration that is currently in use. Then it displays the results of the query. For more information, type man 1 nsquery at the HP-UX prompt.

The syslogd Utility

The syslogd utility logs all the informational and error messages to the file /var/adm/syslog/syslog.log. You can configure syslogd to log the messages to a different destination. See the HP-UX Internet Services Administrator’s Guide at the URL http://www.docs.hp.com /hpux/netcom/index.html#Internet%20Services for more information on syslogd.

Name Server Debugging

By default, all the error messages are logged in the /var/adm/syslog/syslog.log file. You can configure named to log only the name server messages by specifying the debug level when you start named.

To turn on named debugging, issue the following command:

# named -d debug_level

The debugging output is written to the named.run file in the name server’s working directory.

You must use controls to start the name server debugging when named is already running, or if you want to turn off debugging.

To change the debugging level of the name server that is already running at debug level 1, run the following command:

# rndc trace 2

To turn off debugging on a name server that is already running at debug level 2, run the following command:

# rndc notrace

You can also kill the named process and restart named with or without the debugging option. See “Understanding Name Server Debugging Output” for more information on the debugging output. For more information on rndc, see “The rndc Utility”.

Depending on the amount of information the name server provides, BIND offers various debugging levels. The lower the debugging level, the less information is written to the named.run file. Higher debugging levels provide more information, but occupy more disk space. The default debugging level is 0. The debugging information is cumulative, that is, level 2 includes all of level 1’s debugging information, and so on.

BIND provides the following debugging levels:

1: This is the most useful debug level. It logs the basic name server operations: zone loading, maintenance (including SOA queries, zone transfers and zone expiration, and cache cleaning), NOTIFY messages, queries received, and high-level tasks dispatched.
2: This level logs multicast requests.
3: This level logs the low-level task creation and operation and journal activity, such as when the name server writes a record of a zone change to the zone’s journal or when the name server applies a journal to a zone at startup. This level also logs the operation of the DNSSEC validator and checking of TSIG signatures.
4: This level logs when a master name server uses AXFR because the transferred zone’s journal is not available.
5: This level logs the view used while servicing a particular request.
6: This level logs outbound zone transfer messages, including checks of the query that initiated the transfer.
7: This level logs the journals added and deleted, and a count of the number of bytes returned by a zone transfer.
8: This level logs the following dynamic update messages: prerequisite checks, writing journal entries, and rollbacks. This level also logs several low-level zone transfer messages and the resource records sent in a zone transfer.
10: This level logs zone timer activity messages.
20: This level logs updates to a zone’s refresh timer.
90: This level logs low-level operation of the BIND 9 task dispatcher.

At certain debugging levels, the actual packets are displayed. See RFC 1035 (Domain Names – Implementation and Specification) for the format of DNS packets.

Dumping the Name Server Database

You can also track BIND problems by understanding the name server’s internal database dump. You can use the rndc command to dump named’s authoritative data, cached data, and hints data to the file, named_dump.db, in BIND’s working directory.

Issue the following command to generate the dump file, named_dump.db:

# rndc dumpdb

For more information on the rndc command, type man 1M rndc at the HP-UX prompt.

Problem Symptoms

This section describes symptoms of common name server problems, and lists possible problems to check for. A description of the problems appears in “Name Server Problems”.

After configuring the master server for the first time, names in the local domain cannot be found. Check the following:
- Problem 2, Syntax Errors
- Problem 1, Incorrect hosts_to_named Parameters
- Problem 8, Local Domain Not Set
After configuring the master server for the first time, names in the local domain can be found, but names in remote domains fail. Check the following:
- Problem 3, Missing Cache Information
- Problem 5, Network Connectivity
- Problem 7, Incorrect Delegation of Subdomain
After configuring the local host to use a remote server, all name lookups fail, or only names in the NIS database or /etc/hosts are found. The server on the remote host is configured properly. Check the following:
- Problem 4, Syntax Errors in /etc/resolv.conf
- Problem 8, Local Domain Not Set
- Problem 9, /etc/nsswitch.conf Not Configured Correctly
A remote name lookup now fails that has completed successfully before. Check the following:
- Problem 5, Network Connectivity
- Problem 2, Syntax Errors
- Problem 4, Syntax Errors in /etc/resolv.conf
- Problem 10, /etc/hosts or NIS contains Incorrect Data
A local name lookup now fails that has completed successfully before. Check the following:
- Problem 2, Syntax Errors
- Problem 6, Slave Server Unable to Load from Another Slave Server
- Problem 4, Syntax Errors in /etc/resolv.conf
- Problem 5, Network Connectivity
- Problem 10, /etc/hosts or NIS contains Incorrect Data
Names in the local and remote domains are looked up successfully. However, other servers not in your domain cannot look up names within your domain. Check the following:
- Problem 7, Incorrect Delegation of Subdomain

Name Server Problems

This section explains the problems that may cause the symptoms listed above, and suggests ways to solve the problems.

Incorrect parameters supplied to hosts_to_named.
Check the domain data files to be sure they contain records for the hosts in your domain. If localhost is the only host listed, you may have supplied incorrect domain names or network numbers to hosts_to_named.
Syntax error in the named.conf file or a data file.
- syslogd
  Syntax error messages are logged indicating the file name and line number.
- Name server debugging output
  Start the name server at debug level 1. Look for syntax error messages in /var/tmp/named.run indicating the file name and line number.
- ping hostname
  If ping indicates that the host is unknown and the local name server should be authoritative for that name, the syntax error is probably in the file that maps host names to Internet addresses, db.domain.
Missing cache information about the root servers. Without information about the root servers, names outside of the local domain cannot be looked up because the local server relies on the root servers to direct it to servers for other domains.
- syslogd
  Queries for names outside of the local domain cause syslogd to log the following message: No root name servers for class 1. (Class 1 is the IN class.)
- nslookup
  nslookup may fail to look up the local host’s name on startup and give a servfail message. To verify root server information, execute the following:
  $ nslookup
  > set type=NS
  > .
  This asks for the NS records for the root. If no records for root servers are present, it returns Can't find ".": Server failed.
- ping hostname
  Names in the local domain are found, while names in remote domains are not found.
  Name server debugging output
  Set debugging to level 1. Use ping on a host name not in the local domain. The debugging output in /var/tmp/named.run contains the following: No root name servers for class 1. (Class 1 is the IN class.)
  Dumping the name server database
  No root server data appears in the “Hints” section at the end of the file /var/tmp/named_dump.db.
Syntax errors in /etc/resolv.conf (for remote server configuration only). This assumes that the server on the remote host is configured properly. Errors in /etc/resolv.conf are silently ignored by the resolver code.
- ping IP_address or ping hostname
  Only names in the NIS database or /etc/hosts file can be looked up. ping the remote server’s address to verify connectivity.
- Name server debugging output
  Turn on debugging on the remote server. Verify that it is receiving queries from the local host. If queries are not being received, examine the name server entries in /etc/resolv.conf and verify network connectivity to the remote server.

Network connectivity problems may cause certain lookups to fail. See the Installing and Administering LAN/9000 Software manual for information on troubleshooting network connectivity.

Name server debugging output

Turn on debug level 1. Use ping on the host name. Examine the name server debugging output in /var/tmp/named.run for lines like this:

req: found `cucard.med.columbia.edu’ as `columbia.edu’
resend(addr=1 n=0) -> 128.59.32.1 6 (53) nsid=18 id=1 0ms
resend(addr=2 n=0) -> 128.59.40.130 6 (53) nsid=18 id=1 0ms
resend(addr=3 n=0) -> 128.103.1.1 6 (53) nsid=18 id=1 764ms

In this case the name server tries to contact the columbia.edu name servers but does not get a response. Use ping on the addresses the server is trying to contact to verify network connectivity.

If the addresses being tried are the root name servers, either the host does not have connectivity to these machines, or the root server addresses are wrong.

nslookup
nslookup times out while trying to look up the name.
ping hostname
A message is returned saying that the host is unknown.

Slave master is unable to load from another master. This may be caused by a configuration error or problems with network connectivity. Verify that the domain being loaded and the address of the remote server are correct in the configuration file.
- syslogd
  An error message is logged indicating the master server for the secondary zone is unreachable.
- Name server debugging output
  Start the slave server at debugging level 2 or 3. Watch for error messages in the debug output. These could show that the other server is unreachable, the other server is not authoritative for the domain, or the local SOA serial number is higher than the remote SOA serial number for this zone.
- ping IP_address
  Verify connectivity to the server the secondary is trying to load from. If the host is temporarily unreachable, the slave server will load when it is reachable.
- nslookup
  Use nslookup and set the name server to the master the secondary is trying to load from.
  $ nslookup
  > server server_name or IP_address
  > ls domain
  The ls command initiates a zone transfer. If the error message is No response from server, then no server is running on the remote host. If the ls command succeeds, the secondary should be able to load the data from this server.
Incorrect subdomain delegation may be caused by missing or incorrect NS or A records in the parent server for the subdomain.
- nslookup
  Use nslookup to query the parent server for delegation information. Execute the following:
  $ nslookup
  > server parent_server_name or IP_address
  > set type=ns
  > subdomain_name
  This must show you the NS and A records for the subdomain servers, as seen in the following example. In the example, the subdomain is delegated correctly.
  hershey.div.inc.com:rootk> nslookup
  Default Name Server: hershey.div.inc.com
  Addresses: 15.19.14.100, 15.19.15.100
  hershey is the default name server for this host.
  > server eduardo.doc.inc.com.
  Default Name Server: eduardo.doc.inc.com
  Address: 15.19.11.2
  Set the default name server to be this subdomain’s parent server, eduardo.
  > set type=ns
  > div.inc.com
  Name Server: eduardo.doc.inc.com
  Address: 15.19.11.2
  Set query type to ns (name server). Look up the div.inc.com domain.
  Non-authoritative answer:
  div.inc.com nameserver = walleye.div.inc.com
  div.inc.com nameserver = friday.div.inc.com
  Name server records for div.inc.com, the delegated subdomain. Authoritative answers can be found from:
  walleye.div.inc.com inet address = 15.19.13.197
  friday.div.inc.com inet address = 15.19.10.74
  Address records for the name servers for div.inc.com.
- Dumping the name server database
  Because the name server caches information, a database dump can be searched for the NS and A records for the subdomain. If no NS or A records exist, the parent server for the subdomain or the root servers are not reachable. If NS and A records exist, check their correctness. Then try pinging the addresses of the name servers to see if they are reachable.
- Name server debugging output
  Turn on debugging to level 1 and try to look up a name in the domain. Verify the debug output for name server retransmissions. This indicates which servers are not responding. Verify that the servers and their addresses are correct, if possible.
The local domain is not set. The local domain is used to complete names that do not end with a dot. To set the local domain, either set the host name (hostname) of the local system to a domain name (without a trailing dot), or add a domain entry to /etc/resolv.conf.
- nslookup
  nslookup gives a warning that the local domain is not set.
- Name server debugging output
  The debug output at level 1 shows names being looked up that are not domain names.
- ping hostname
  hostname is found only when it is a completely specified domain name (with or without a trailing dot).
The /etc/nsswitch.conf file, if it exists, is not configured correctly. If you want to query BIND before querying NIS or the /etc/hosts file, make sure dns is listed first on the hosts line. See the HP-UX Internet Services Administrator’s Guide for information on how to configure the name service switch.
The /etc/hosts file or NIS contains incorrect data. The name service switch (/etc/nsswitch.conf) allows host name lookups in /etc/hosts or NIS, and one of those databases contains incorrect data. For information on configuring the /etc/hosts file, see the HP-UX Internet Services Administrator’s Guide for information on how to edit the /etc/hosts file. For information on NIS, see NIS Administrator's Guide.

Understanding Name Server Debugging Output

To diagnose problems in the debugging output of the name server, you need to know what output from a successful query looks like. The following two examples show output from successful host name lookups. The first example does not involve any retransmissions, while the second example does. Note that debugging output looks the same whether it comes from a master, slave, or caching-only server.

Example 1: No Retransmissions

Debug turned ON, Level 1

datagram from 15.19.10.14 port 4258, fd 6, len 35
req: nlookup(john.dept.inc.com) id 1 type=1
req: found ‘john.dept.inc.com’ as ‘inc.com’ (cname=0)
forw: forw -> 192.67.67.53 6 (53) nsid=29 id=1 0ms retry 4 sec

datagram from 192.67.67.53 port 53, fd 6, len 166
resp: nlookup(john.dept.inc.com) type=1
resp: found ‘john.dept.inc.com’ as ‘inc.com’ (cname=0)
resp: forw -> 15.19.11.2 6 (53) nsid=32 id=1 0ms

datagram from 15.19.11.2 port 53, fd 6, len 119
resp: nlookup(john.dept.inc.com) type=1
resp: forw -> 15.19.15.15 6 (53) nsid=33 id=1 0ms

datagram from 15.19.15.15 port 53, fd 6, len 51
send_msg -> 15.19.10.14 (UDP 7 4258) id=1
Debug turned OFF, Level 1

In the first group of four lines, a query is received for john.dept.inc.com. The query is forwarded to a root server, ns.inc.ddn.mil, at address 192.67.67.53
In the second group of four lines, ns.nic.ddn.mil responds with NS records for inc.com.
In the third group of four lines, the inc.com server responds with NS records for dept.inc.com.
In the fourth group of four lines, the dept.inc.com server responds with the address of john. The local server responds with the answer to 15.19.10.14.

Following are detailed explanations of certain lines from the previous example.

Debug turned ON, Level 1

The name server was already running. The first level of debugging was turned on with sig_named debug 1.

datagram from 15.19.10.14 port 4258, fd 6, len 35

This line shows the IP address of the host that generated the query, the port that the request comes from, the file descriptor that the name server received the query on, and the length of the query.

req: nlookup(john.dept.inc.com) ID 1 type=1

This message was logged from the routine that handles requests. Shown are the name looked up, the packet ID (used to determine duplicate requests), and the type (as defined in /usr/include/arpa/nameser.h). Type 1 is an address query.

req: found 'john.dept.inc.com' as 'inc.com' (cname=0)

Because the server is authoritative for div.inc.com, it has an entry for inc.com in its database. The only data at inc.com is the subdomain entry for div. This line does not indicate what was found at inc.com. Because the server sent the next query to a root name server, you can conclude that there were no NS records for inc.com. For more information, including the domain for which the queried server is authoritative, check Debug level 3. Debug levels are explained in “Name Server Debugging”.

forw: forw -> 192.67.67.53 6 (53) nsid=29 id=1 0ms retry 4 sec

The query was forwarded to 192.67.67.53. The name server tags each query it sends out so that it can detect duplicate responses. Here the assigned ID is 29. The original ID was 1. The query will be retried in four seconds.

resp: found 'john.dept.inc.com' as 'inc.com' (cname=0)

After the response from the root server, the database is searched again. inc.com is found once again. The next query goes to an inc.com server, so this time there were NS records.

datagram from 15.19.11.2 port 53, fd 6, len 119

This datagram is from another name server because it is from port 53. Because the example server sent a query to 15.19.11.2, you can assume this is the response.

send_msg -> 15.19.10.14 (UDP 7 4258) id=1

The response was sent back to host 15.19.10.14 on port 4258.

Example 2: Retransmissions

The next example shows a successful lookup that involves retransmissions. Retransmissions take place from the resolver and the name server. The resolver retransmits to the local name server, and the local name server retransmits to remote name servers during the process of looking up a name. When the local server receives the resolver retransmissions, it discards them as duplicates if it is still processing the first request.

datagram from 15.19.10.14 port 4253, fd 6, len 41
req: nlookup(cucard.med.columbia.edu) id 1 type=1
req: found ‘cucard.med.columbia.edu’ as ‘edu’ (cname=0)
forw: forw -> 128.9.0.107 6 (53) nsid=17 id=1 1478ms retry 4 sec

datagram from 128.9.0.107 port 53, fd 6, len 212
resp: nlookup(cucard.med.columbia.edu) type=1
resp: found ‘cucard.med.columbia.edu’ as ‘columbia.edu’ (cname=0)
resp: forw -> 128.59.16.1 6 (53) nsid=18 id=1 0ms

datagram from 15.19.10.14 port 4253, fd 6, len 41
req: nlookup(cucard.med.columbia.edu) id 1 type=1
req: found ‘cucard.med.columbia.edu’ as ‘columbia.edu’ (cname=0)
resend(addr=1 n=0) -> 128.59.32.1 6 (53) nsid=18 id=1 0ms
resend(addr=2 n=0) -> 128.59.40.130 6 (53) nsid=18 id=1 0ms

datagram from 15.19.10.14 port 4253, fd 6, len 41
req: nlookup(cucard.med.columbia.edu) id 1 type=1
req: found ‘cucard.med.columbia.edu’ as ‘columbia.edu’ (cname=0)
resend(addr=3 n=0) -> 128.103.1.1 6 (53) nsid=18 id=1 764ms

datagram from 128.103.1.1 port 53, fd 6, len 57
send_msg -> 15.19.10.14 (UDP 7 4253) ID=1

Following are detailed explanations of certain lines from this example.

req: nlookup(cucard.med.columbia.edu) id 1 type=1

This message is logged from the routine that handles requests. The following shows the name looked up, the packet ID (used to determine duplicate requests), and the type (as defined in /usr/include/arpa/nameser.h). Type 1 is an address query.

resend(addr=1 n=0) -> 128.59.32.1 6 (53) nsid=18 id=1 0ms

Because no response is received from 128.59.16.1, the query with nsid 18 is resent to other servers.

datagram from 15.19.10.14 port 4253, fd 6, len 41
 req: nlookup(cucard.med.columbia.edu) id 1 type=1

This is received from the same IP address and port and has the same length and ID as the preceding datagram. It is a duplicate and thus forw discards it. These two lines are repeated three times throughout the trace. The queries came from the same IP address and port, and have the same ID and length in each case. Thus, these are all the same query. The resolver sends the query three times because the name server did not respond. The name server detects that the second and third are duplicates and discards them because the duplicates did not get to the forw line.

Name Server Statistics

The name server keeps track of various statistics. You can print these statistics to the file /var/tmp/named.stats by issuing the following command:

/usr/sbin/sig_named stats

Statistics are appended to the file. The statistics look similar to this:

`1273431`	`time since boot (secs)`
`29802`	`time since reset (secs)`
`326031`	`input packets`
`327165`	`output packets`
`284353`	`queries`
`0`	`iqueries`
`214`	`duplicate queries`
`50109`	`responses`
`70`	`duplicate responses`
`220220`	`OK answers`
`63919`	`FAIL answers`
`0`	`FORMERR answers`
`23`	`system queries`
`4`	`prime cache calls`
`4`	`check_ns calls`
`0`	`bad responses dropped`
`0`	`martian responses`
`0`	`Unknown query types`
`47921`	`A queries`
`2054`	`CNAME queries`
`8216`	`SOA queries`
`35909`	`PTR queries`
`10569`	`MX queries`
`424`	`AXFR queries`
`179263`	`ANY queries`

The first two lines print out the number of seconds that the name server has been running and the number of seconds since the last restart caused by a SIGHUP signal. To convert these values to days, divide by 86,400 (the number of seconds in a day).

input packets is the number of datagrams received by the name server. The datagrams come from the resolver code compiled into the services and from queries and responses from other name servers.

output packets is the number of datagrams sent by the name server. These datagrams are responses to resolver queries, queries from other name servers, and system queries. Because queries to other name servers may not be answered, there will probably be more output packets than input packets.

queries is the number of queries received by this name server. Because the name server can handle datagram and stream connections, there can be more queries than input packets. The total number of queries is the sum of all the counts of different query types listed in this statistics dump, starting with unknown query types.

iqueries is the number of inverse queries. Inverse queries can be used to map a host address to a domain name, although PTR queries are the normal method. Some versions of nslookup send inverse queries when they are starting up.

duplicate queries are retransmitted queries for pending lookups that the resolver sends to the name server. The name server detects the duplicate queries and discards them.

responses is the number of response packets that the name server receives from queries to other name servers.

duplicate responses are response packets from remote name servers for queries that are no longer pending. The name server retransmits queries to remote name servers. If the remote server responds to the original query and responds to the retransmitted query, the local name server discards the second response as a duplicate.

OK answers is the number of responses to queries that contain some information.

FAIL answers is the number of responses indicating either that the name does not exist or that there is no data of the requested type for this name.

FORMERR answers is the number of malformed response packets from other name servers. A message is sent to the syslog daemon listing the sender of the malformed response packet.

system queries are queries generated by the name server. These usually occur when the name server detects another name server listed for a domain for which there is no address data. The system query is an attempt to find the address data for that name server. System queries are also used to keep up-to-date information about the name servers for the root domain.

prime cache calls are calls to update the information about the name servers for the root domain.

check_ns calls are calls to check the state of the information about the name servers for the root domain.

bad responses dropped are responses from remote name servers that are dropped. These occur most often when the remote name server responds with SERVFAIL, indicating a problem with the server’s domain data.

martian responses are responses from unexpected addresses. The name server keeps track of how long it takes for a remote name server to respond. If the remote name server is a multi-homed host, a query to one of the addresses may result in a response from another of its addresses. If the local server does not know about this other address, the response is counted as a martian response.

unknown query types are queries for data types unknown to this server.

A queries are queries for the host address for a domain name. The gethostbyname library routine generates these address queries.

CNAME queries are queries for the canonical name for a domain name. Some versions of sendmail query for CNAME records during name canonicalization from $[$] tokens in /var/adm/sendmail/sendmail.cf.

SOA queries are queries for the start of authority records. These queries are most often made by slave servers over a stream connection to check if their domain data is current.

PTR queries are queries for the domain name for a host address. The gethostbyaddr library routine generates these queries.

MX queries are mail exchanger queries made by sendmail during the delivery of electronic mail.

AXFR queries are the number of zone transfers done by slave servers. A slave server first makes an SOA query and follows that with an AXFR query if new domain data should be transferred.

ANY queries are queries for any data associated with the domain name. Some versions of sendmail make queries for ANY data during name canonicalization from $[$] tokens in /var/adm/sendmail/sendmail.cf.