Troubleshooting Scenarios

IPSec is incorrectly allowing packets to pass in clear text instead of authenticating, encrypting, or discarding the packets.

No error message or interruptions to user service, but no SAs are established, or IPSec is passing packets that should be discarded to upper layers.

Run the following commands:

ipsec_report -sad (check for IPSec/QM SAs)

ipsec_policy (determine the policy being used)

ipsec_report -cache (check the cached policy decisions)

ipsec_report -host (check for active host IPSec policies)

ipsec_report -bypass (verify that the local address is not in the bypass list)

Check the configuration file for incorrect addresses, order, or other incorrect information.

If HP-UX IPSec is misconfigured to pass packets that it should authenticate or encrypt, there will be no obvious external symptoms. Check if HP-UX IPSec actually established SAs and is encrypting/authenticating the packets. Check for IPSec/QM SAs using the following commands:

ipsec_report -sad
ipsec_report -host

If there are no SAs for the IP packets that you expect and no user error, HP-UX IPSec is probably misconfigured and passing packets it should not. Check to see which IPSec policy is being used by running ipsec_policy, or by executing the ipsec_report -cache and ipsec_report -host commands.

Verify that the local IPv4 address is not in the bypass list (ipsec_report -bypass).

IPSec is attempting to encrypt or authenticate (apply a transform) packets that should not be encrypted or authenticated.

Link errors (unable to connect or connection timeouts) on traffic that should not be encrypted/authenticated.

Run the following commands:

ping, linkloop (check connectivity)

ipsec_policy or ipsec_report -cache and ipsec_report -host (determine the policy being used)

Check the configuration file.

If HP-UX IPSec is misconfigured to encrypt and/or authenticate packets that it should not and the peer system is not configured to use HP-UX IPSec encryption/authentication, you will consistently get connection errors (unable to connect or connection timed out).

Check connectivity to the remote system using /etc/ping and the linkloop utilities.

Verify which IPSec policy is being used with the ipsec_policy command and check the configuration file.

IPSec attempts to encrypt/authenticate packets and fails.

Link errors (unable to connect) and ipsec_report -sad shows no IPSec/QM SAs.

Determine if ISAKMP/MM SA negotiations are succeeding. Run the following commands:

ipsec_report -mad
ipsec_report -audit file

Check for Main Mode processing failed, MM negotiation timeout error messages in the log file.

If HP-UX IPSec is configured to encrypt/authenticate but failing, it will appear as a connection error (unable to connect or connection timed out) to the user.

If users are consistently getting connection errors for traffic that should use HP-UX IPSec for encryption or authentication, check for IPSec/QM SAs using the following commands:

ipsec_report -sad
ipsec_report -host

Determine if IPSec is successfully creating the ISAKMP/MM SA. Check for ISAKMP/MM SAs using the following command:

ipsec_report -mad

If there is no ISAKMP/MM SA, HP-UX IPSec may have created an ISAKMP/MM SA but deleted it when the IPSec/QM SA negotiation failed. Check the audit log for failed attempts to establish ISAKMP/MM SAs using the following command:

ipsec_report -audit /var/adm/ipsec/auditdateinfo.log

Check the log file for IKMPD Main Mode processing failed error entries such as the following:

Msg: 31 From: IKMPD LVL: ERROR Date: Wed Oct 31 11:44:10 2001
Event: Main Mode processing failed

Also check the log file for MM negotiation timeout error entries such as the following:

Msg: 413 From: IKMPD Lvl: ERROR Date: Fri Mar 15 07:14:18 2002
Event: MM negotiation timeout, src 15.2.2.2

If there is a mismatch in IKE policies, some IKE daemons do not respond to negotiation attempts. This causes a MM negotiation timeout error on the connecting system.

ISAKMP/MM SA negotiation fails.

The output from ipsec_report -mad output does not show the ISAKMP/MM SA. The audit log contains a Main Mode processing failed or MM negotiation timeout error entry.

Determine whether the ISAKMP/MM SA is absent because the ISAKMP/MM negotiation failed or because the successfully negotiated ISAKMP/MM SA was deleted when an IPSec/QM negotiation failed.

Run the following commands:

ipsec_admin -auditlvl informative (or debug)

ipsec_report -audit audit_file_name [-entity ikmpd]

ipsec_admin trace (check for packets to and from UDP port 500)

If there is no ISAKMP/MM SA to the remote system, the ISAKMP/MM SA negotiation may be failing.

If IPSec/QM negotiations fail, the remote IKE sends the HP-UX IKE daemon notification that the negotiation failed. The HP-UX IKE daemon then notifies the peer IKE daemon that it wants to delete the ISAKMP/MM SA that was used for the failed IPSec/QM negotiation.

Determine whether or not the ISAKMP/MM SA is being established by checking the audit log file. IKMPD error entries with the message Main Mode processing failed indicate that the ISAKMP/MM SA is not being established. Informative entries with the message MM negotiation complete with the peer ip_address indicate that the ISAKMP/MM SA is being established.

The IKMPD message QM negotiation timeout, mess ID hhhh may indicate that there is an IPSec transform proposal mismatch (the IKE peer may not respond if it receives an unacceptable transform proposal, which causes a timeout).

If the ISAKMP/MM negotiation was successful but the ISAKMP/MM SA was deleted later because an IPSec/QM negotiation failed, go on to “ISAKMP/MM SA Negotiation Succeeded, IPSec/QM SA Negotiation Fails (Quick Mode processing failed, QM negotiation timeout)”.

If the ISAKMP/MM SA is not being established:

Run the following command:

ipsec_policy (determine the IKE policy)

Check the IKE policy parameters against the parameters configured on the remote system:

Check the audit file for IKMPD error messages such as MM negotiation timeout. This may indicate a connectivity problem with the remote system. However, some ISAKMP/IKE responders will not respond if the initiating system sends an unacceptable SA proposal, which also causes timeouts.

Enable a nettl level 4 trace using the command ipsec_admin -traceon or get a line analyzer trace and verify that the packets are being sent and received by the correct remote system. Check whether the remote ISAKMP/IKE entity is responding. ISAKMP always uses UDP port 500 to receive and send ISAKMP packets.

ISAKMP primary authentication with preshared key fails.

Output from the ipsec_report -mad command does not show the ISAKMP/MM SA. The audit log contains a Main Mode process failed message.

Verify that the preshared key values match. Use the ipsec_config show auth command to verify the preshared key configured on the local system. Check the key format on the remote system (ASCII or hex); HP-UX IPSec always configures preshared keys as ASCII values. Check the audit file.

Certificate-based (RSA signature) primary authentication fails.

Output from the ipsec -mad command does not show the ISAKMP/MM SA. The audit log contains a Main Mode processing failed error message.

Check the audit file for an expired certificate, revoked certificate, or certificate encoding problems. Try preshared key authentication.

Run ipsec_mgr and check for a certificate for the remote system in /var/adm/ipsec/certs.txt (VeriSign) or /var/adm/ipsec/.Bcerts (Baltimore).

Check for the /var/adm/ipsec/javabeans.txt (VeriSign) or /var/adm/ipsec/.Bsec file (Baltimore).

Check the audit log for messages indicating that the certificate for the local or remote system has expired, has been revoked, or has X.509 encoding errors.

You can also try using preshared keys for primary authentication. You will need to configure the same preshared key on both systems.

Check that you have a certificate for the remote system. As part of the IKE dialog, the remote system should send its certificate to the local system. The IKE daemon stores a copy of the certificate in /var/adm/ipsec/certs.txt (VeriSign) or /var/adm/ipsec/.Bcerts (Baltimore). However, these files are encrypted and can only be viewed with ipsec_mgr. Check the expiration date for the local and remote system certificates.

Check that the /var/adm/ipsec/javabeans.txt file (VeriSign) or the /var/adm/ipsec/.Bsec file (Baltimore) has not been deleted. If the applicable file has been deleted, either restore it from a backup or recreate it by re-importing the certificate.

For VeriSign, check that the entry in the certs.txt file for the local system is complete by using ipsec_mgr to examine the certificates in detail. If you have requested a VeriSign certificate but have not completed the process of importing the certificate into IPSec, you will find an entry in the /var/adm/ipsec/certs.txt or /var/adm/ipsec/.Bcerts file for the local system, but there will be no certificate.

ISAKMP/MM SA negotiation succeeded, the ISAKMP/MM SA was established, but the IPSec/QM SA negotiation failed.

Output from the ipsec_report -sad command does not show IPSec/QM SAs and the audit log contains Quick Mode processing failed or QM negotiation timeout error messages.

Run ipsec_policy to determine the IPSec policy that HP-UX IPSec is using, or execute the ipsec_report -cache and ipsec_report -host commands.

Check the transform list and lifetimes. Check the audit file.

If the ISAKMP/MM SA negotiation succeeded but the IPSec/QM SA negotiation failed, you will probably not see any ISAKMP/MM SAs in the output of the ipsec_report -mad command. This is because the HP-UX IPSec IKE daemon tears down an ISAKMP/MM SA if an IPSec/QM SA negotiation fails. To be sure that the ISAKMP/MM negotiation succeeded and that IKE actually attempted to negotiate the IPSec/QM SA, look for Quick Mode processing failed or QM negotiation timeout error messages in the audit file. A QM negotiation timeout error usually indicates that the remote system did not agree with the IPSec/QM SA proposal and chose not to respond.

Check which IPSec policy is being used with the ipsec_policy command. Check the IPSec policy configurations for mismatches.

Manual keys do not work.

Link errors (unable to connect) and timeouts. The output from the ipsec_report -sad command shows the SAs, but attempts to exchange data with the remote system fail.

The audit file may show errors when HP-UX IPSec starts or when a manual key is added such as PF_KEY: SADB_ADD for SPI 0xnnnn returns EEXIST and PF_KEY: Invalid SADB_ADD, SPI 0xnnnn, errno 22.

If the HP-UX IPSec audit level is set to warning or higher, the audit file may show entries such as No SPI for received packet.

STREAMS log records may show entries from HP-UX IPSec STREAMS modules (ipsec_aaaa), such as Bad cipher SA init or Padding checks failed.

Check the manual key configuration. Check that the local inbound SA descriptor (SPI, transform type, authentication key, encryption key, and initialization vector) matches the remote outbound SA descriptor (on the remote system). Check that the local outbound SA descriptor matches the remote inbound SA descriptor (on the remote system).

If the audit file contains the error message similar to the one below, HP-UX IPSec may be rejecting a DES or 3DES encryption key because it is too weak (not sufficiently random):

PF_KEY: Invalid SADB_ADD, SPI 0xnnnn, errno 22

Verify that the SPI number in the audit message matches a manual key SPI. Examine the STREAMS log messages to verify that the error is caused by a weak encryption key, as described in “Examining STREAMS Logging Records”. See Chapter 7 “HP-UX IPSec and HP-UX Mobile IPv6”, “Selecting Encryption Keys” for information on generating strong encryption keys.

In most cases, little information is logged when manual keys fail because there is no IKE or IPSec SA negotiation. The ipsec_report -sad and ipsec_report -host active output show the SAs when the SA information is added to the runtime database, even if the SAs are not acceptable to the remote system. To view additional data that may include information about manual key SAs, use the following procedures to examine the STREAMS logging records and additional audit file entries.

HP-UX IPSec will not start.

The ipsec_admin -start command fails. The ipsec_admin utility returns one of the following messages:

IPSEC_ADMIN: Failed to read IPSec admin file, error: %nn. Did you set the password with -np?

IPSEC_ADMIN: Failed to open IPSec admin file, error: %nn. Did you set the password with -np?

IPSEC_ADMIN: ERROR-read_admin_info(): Failed to verify ipsec password.

IPSEC_ADMIN: ERROR-reads a DB config which is invalid

IPSEC_ADMIN: ERROR-Configuration database open failed: reason

If ipsec_admin returns the message Failed to read IPSec admin file, error: %nn. Did you set the password with -np? or the message Failed to open IPSec admin file, error: %nn. Did you set the password with -np? and you have not yet set the HP-UX IPSec password, set the password using the command ipsec_admin -newpasswd or ipsec_admin -np. Verify that the file /var/adm/ipsec/.admin_info exists. If this file not exist, restore it or use the procedure described in the section “Re-establishing the HP-UX IPSec Password” to re-establish the password

If ipsec_admin returns the message read_admin_info(): Failed to verify ipsec password, verify that the file /var/adm/ipsec/.admin_info exists. If this file does not exist, restore it or use the procedure described in the section “Re-establishing the HP-UX IPSec Password” to re-establish the password.

If ipsec_admin returns the message reads a DB config which is invalid or Configuration database open failed, see the following section, “Corrupt or Missing Configuration Database”, for more information.

The configuration database file (/var/adm/ipsec/config.db) is corrupt or missing.

The symptom vary according to when the problem is detected. HP-UX IPSec modules will log error messages to the audit log file and user utilities will also display the error messages to stdout.

If ipsec_admin detects the problem (for example, when the user is executing the ipsec_admin -start command), ipsec_admin logs and displays one of the following messages:

IPSEC_ADMIN: ERROR-reads a DB config which is invalid

IPSEC_ADMIN: ERROR-Configuration database open failed: reason

If ipsec_config detects the problem, ipsec_config logs and displays a message similar to one of the following messages:

If the policy daemon detects that configuration database is corrupted, the policy daemon logs an error message similar to the following:

Msg: 413 From: SECPOLICYD Lvl: ERROR Date: Sun May 09 10:21:32 2004
Event: /var/adm/ipsec/config.db file is corrupt.

Re-create or restore the configuration database file (/var/adm/ipsec/config.db). There are two methods to do this:

You can only use this method if you still have a configuration file from a previous release.

Use this method if you do not have a configuration file from a previous version.

Autoboot fails.

HP-UX IPSec does not start automatically at system boot-up time.

Use the following procedure:

If you still have problems after following the troubleshooting procedure, contact your HP representative.

If HP-UX IPSec is not using the IPSec policy you expected, check for errors in the configuration file, such is incorrect IP addresses. Check the order of the IPSec policies—HP-UX IPSec sequentially searches the IPSec policies and selects the first policy with filter parameters that match the packet.

The administrator cannot get a local VeriSign certificate.

The administrator cannot get a certificate using ipsec_mgr. ipsec_mgr shows no certificate entry or an incomplete entry. The audit file shows the following error: Unable to obtain public/private key pair!

Check stdout for ipsec_mgr errors. Check the VeriSign Managed PKI Control Center for a pending request or existing certificate. Check the web proxy configuration (run ipsec_mgr, click on the Options menu, select System, then select Proxy Information). You may need to reset /var/adm/ipsec/cainfo.txt; remove /var/adm/ipsec/javabeans.txt, /var/adm/ipsec/certs.txt; revoke the old certificate, and restart the registration procedure.

If you cannot get a VeriSign certificate or if you forget to retrieve the certificate after you request it, ipsec_mgr may show a certificate entry for the local system. However, if you click Details the entry will be incomplete (there will be no information other than the IP address). When you attempt to initiate an ISAKMP/MM SA, it will fail and you will see errors similar to the following in the log file:

Msg: 201 From: IKMPD Lvl: ERROR Date: Mon Mar 11 16:19:39 2002
 Event: Unable to obtain public/private key pair!
Msg: 202 From: IKMPD Lvl: ERROR Date Mon Mar 11 16:19:39 2002
 Event: Process contruct error 0x1
Msg: 203 From IKMPD Lvl: ERROR Date Mon Mar 11 16:19:39 2002
 Event: Main Mode processing failed

If ipsec_mgr has problems retrieving a VeriSign certificate, it will write errors to the stdout device for ipsec_mgr (the device from which ipsec_mgr was started). The ipsec_mgr does not log errors using the HP-UX IPSec daemon.

If you request a VeriSign certificate using ipsec_mgr and ipsec_mgr cannot retrieve the certificate (the Check on Request operation fails), it is possible that the Managed PKI Administrator has not yet approved your certificate request. Have the Managed PKI Administrator use the VeriSign Managed PKI Control Center to check for pending requests. If the Managed PKI Administrator did not receive a request for approval, verify communication with the Managed PKI Control Center. If you are using a web proxy server to access the VeriSign Managed PKI Control Center, verify the proxy server configuration (run ipsec_mgr, click on the Options menu, select System, then select Proxy Information).

Have the Managed PKI Administrator use the View Certificates area of the Managed PKI Control Center to check for an existing certificate. If a certificate already exists for your system, the VeriSign CA rejects any requests for a new certificate.

Restarting Registration

If VeriSign registration fails, the associated files may be left in an unusable state. You must reset them before trying to reregister. To do this, change the pending field in /var/adm/ipsec/cainfo.txt to false and verify that the VeriSign domain name and CA address are correct, as follows:

begin verisign pending: false domain: name_of_domain caserver: VeriSign_CA_address

You must also delete the /var/adm/ipsec/javabeans.txt and /var/adm/ipsec/certs.txt files.

Check the Managed PKI Control Center for an already existing certificate request or certificate, and deny or revoke it before restarting the registration process.

The Security Policy Database (SPD) is near or exceeding the soft or hard size limit.

The SPD is the HP-UX IPSec runtime policy database, with cached policy decisions for packet descriptors (five-tuples consisting of exact, non-wildcard source IP address, destination IP address, protocol, source port, and destination port).

When the size of the SPD exceeds the soft limit, HP-UX IPSec logs an alert message to the system console and the audit file, and logs an additional alert message for each 1000 SPD entries added. You will see log messages are similar to the following:

where nnnn is the soft limit.

When the hard limit is exceeded, HP-UX IPSec stops adding new entries to the SPD and stops transmitting and receiving packets that do not match existing entries in the SPD. You will see log messages are similar to the following:

where nnnn is the hard limit.

Use the following ipsec_config commands to set and configure new SPD soft and hard limits:

ipsec_config add startup -spd_soft spd_soft_limit

ipsec_config add startup -spd_hard spd_hard_limit

The spd_soft_limit and spd_hard_limit are specified in units of 1000 entries. Refer to the ipsec_config(1M) manpage for more information.

You can also use the ipsec_admin -spd_soft spd_soft_limit and ipsec_admin -spd_hard spd_hard_limit commands to set new SPD soft and hard limits. Refer to the ipsec_admin(1M) manpage for more information.