Troubleshooting Scenarios

Autoboot fails.

HP-UX IPSec does not automatically start at system boot-up time.

Use the following procedure:

If you still have problems after following the troubleshooting procedure, contact your HP representative.

If HP-UX IPSec is not using the IPSec policy you expected, check for errors in the configuration file, such is incorrect IP addresses. Check the order of the IPSec policies—the order affects how HP-UX IPSec selects policies. See “How HP-UX IPSec Selects Policies” for more information.

IPSec is incorrectly allowing packets to pass through instead of authenticating, encrypting, or discarding the packets.

No error message or interruptions to user service, but no SAs are established, or IPSec is passing packets that should be discarded to upper layers.

Run the following commands:

ipsec_report -policy (check for IPSec/QM SAs)

ipsec_report -sad (check for IPSec/QM SAs)

ipsec_policy (determine the policy being used)

Check the configuration file.

If HP-UX IPSec is misconfigured to pass packets that it should authenticate or encrypt, there will be no obvious external symptoms. Check that HP-UX IPSec actually established SAs and is encrypting/authenticating what you want. Check for IPSec/QM SAs using the following commands:

ipsec_report -sad
ipsec_report -policy

If there are no SAs for the IP packets that you expect and no user error, HP-UX IPSec is probably misconfigured and passing packets it should not. Check to see which IPSec policy is being used by running ipsec_policy. You can also use the cookie from the ipsec_report -cache entry for the packet to find the matching ipsec_report -policy entry.

Check the configuration file for incorrect addresses, order, or other incorrect information.

IPSec is attempting to encrypt or authenticate (apply a transform) packets that should not be encrypted or authenticated.

Link errors (unable to connect) on traffic that should not be encrypted/authenticated.

Run the following commands:

ping, linkloop (check connectivity)
ipsec_policy (determine the policy being used)
Check the configuration file.

If HP-UX IPSec is misconfigured to encrypt and/or authenticate packets that it should not and the peer system is not configured to use HP-UX IPSec encryption/authentication, you will consistently get connection errors (unable to connect or connection timed out).

Check connectivity to the remote system using /etc/ping and the linkloop utilities.

Verify which IPSec policy is being used with the ipsec_policy command and check the configuration file.

IPSec attempts to encrypt/authenticate packets and fails.

Link errors (unable to connect) and ipsec_report -sad shows no IPSec/QM SAs.

Determine if ISAKMP/MM SA negotiations are succeeding. Run the following commands:

ipsec_report -mad
ipsec_report -audit file

Check for Main Mode processing failed, MM negotiation timeout error messages in the log file.

If HP-UX IPSec is configured to encrypt/authenticate but failing, it will appear as a connection error (unable to connect or connection timed out) to the user.

If users are consistently getting connection errors for traffic that should use HP-UX IPSec for encryption or authentication, check for IPSec/QM SAs using the following commands:

ipsec_report -sad
ipsec_report -policy

Determine if IPSec is successfully creating the ISAKMP/MM SA. Check for ISAKMP/MM SAs using the following command:

ipsec_report -mad

If there is no ISAKMP/MM SA, HP-UX IPSec may have created an ISAKMP/MM SA but deleted it when the IPSec/QM SA negotiation failed. Check the audit log for failed attempts to establish ISAKMP/MM SAs using the following command:

ipsec_report -audit /var/adm/ipsec/auditdateinfo.log

Check the log file for Main Mode processing failed error entries such as the following:

Msg: 31 From: IKMPD LVL: ERROR Date: Wed Oct 31 11:44:10 2001
Event: Main Mode processing failed

Also check the log file for MM negotiation timeout error entries such as the following:

Msg: 413 From: IKMPD Lvl: ERROR Date: Fri Mar 15 07:14:18 2002
Event: MM negotiation timeout, src 15.2.2.2

If there is a mismatch in ISAKMP policies, some IKE daemons simply do not respond to negotiation attempts. This causes a MM negotiation timeout error on the connecting system.

ISAKMP/MM SA negotiation fails.

The output from ipsec_report -mad output does not show the ISAKMP/MM SA. The audit log contains a Main Mode processing failed or MM negotiation timeout error entry.

Determine whether the ISAKMP/MM SA is absent because the ISAKMP/MM negotiation failed or because the successfully negotiated ISAKMP/MM SA was deleted when an IPSec/QM negotiation failed.

Run the following commands:

ipsec_admin -auditlvl informative (or debug)

ipsec_report -audit audit_file_name

ipsec_admin trace (check for packets to and from UDP port 500)

If there is no ISAKMP/MM SA to the remote system, the ISAKMP/MM SA negotiation may be failing.

If IPSec/QM negotiations fail, the remote IKE sends the HP-UX IKE daemon notification that the negotiation failed. The HP-UX IKE daemon then notifies the peer IKE daemon that it wants to delete the ISAKMP/MM SA that was used for the failed IPSec/QM negotiation.

Determine whether or not the ISAKMP/MM SA is being established by checking the audit log file. Error entries with the message Main Mode processing failed indicate that the ISAKMP/MM SA is not being established. Informative entries with the message MM negotiation complete with the peer ip_address indicate that the ISAKMP/MM SA is being established.

The message QM negotiation timeout, mess ID hhhh may indicate that there is an IPSec transform proposal mismatch (the IKE peer may not respond if it receives an unacceptable transform proposal, which causes a timeout).

If the ISAKMP/MM negotiation was successful but the ISAKMP/MM SA was deleted later because an IPSec/QM negotiation failed, go on to “ISAKMP/MM SA Negotiation Succeeded, IPSec/QM SA Negotiation Failed (Quick Mode processing failed, QM negotiation timeout)”.

If the ISAKMP/MM SA is not being established:

Run the following command:

ipsec_policy (determine ISAKMP policy)

Check which ISAKMP policy is being used with the ipsec_policy utility. The ISAKMP policy actually used may not be the same ISAKMP policy specified in the IPSec policy, so be sure to check this in the ipsec_policy output.

Check the ISAKMP policy parameter against the remote systems parameters:

Check the audit file for error messages such as MM negotiation timeout. This may indicate a connectivity problem with the remote system. However, some ISAKMP responders will not respond if the initiating system sends an unacceptable SA proposal, which will also cause a timeout.

Enable a nettl level 4 trace using the command ipsec _admin -traceon or get a line analyzer trace and verify that the packets are being sent and received by the correct systems. Check whether the remote ISAKMP/IKE entity is responding. ISAKMP always uses UDP port 500 to receive and send ISAKMP packets.

ISAKMP primary authentication with preshared key fails.

Output from ipsec_report -mad does not show the ISAKMP/MM SA. The audit log contains a Main Mode process failed message.

Verify that the preshared key values match. Check the key format on the remote system (ASCII or hex); HP-UX IPSec always configures preshared keys as ASCII values. Check the audit file.

Certificate-based (RSA signature) primary authentication fails.

Output from ipsec -mad does not show the ISAKMP/MM SA. The audit log contains a Main Mode processing failed error message.

Check the audit file for an expired certificate, revoked certificate, or certificate encoding problems. Try preshared key authentication.

Run ipsec_mgr and check for a certificate for the remote system in /var/adm/ipsec/certs.txt (VeriSign) or /var/adm/ipsec/.Bcerts (Baltimore).

Check for the /var/adm/ipsec/javabeans.txt (VeriSign) or /var/adm/ipsec/.Bsec file (Baltimore).

Check the audit log for messages indicating that the local or remote system’s certificate has expired, has been revoked, or has X.509 encoding errors.

You can also try using preshared keys for primary authentication. You will need to configure the same preshared key on both systems.

Check that you have a certificate for the remote system. As part of the IKE dialog, the remote system should send its certificate to the local system. The IKE daemon stores a copy of the certificate in /var/adm/ipsec/certs.txt (VeriSign) or /var/adm/ipsec/.Bcerts (Baltimore). However, these files are encrypted and can only be viewed with ipsec_mgr. Check the expiration date for the local and remote system certificates.

Check that the /var/adm/ipsec/javabeans.txt file (VeriSign) or the /var/adm/ipsec/.Bsec file (Baltimore) has not been deleted. If the applicable file has been deleted, either restore it from a backup or recreate it by re-importing the certificate.

For VeriSign, check that the entry in the certs.txt file for the local system is complete by using ipsec_mgr to examine the certificates in detail. If you have requested a VeriSign certificate but have not completed the process of importing the certificate into IPSec, you will find an entry in the /var/adm/ipsec/certs.txt or /var/adm/ipsec/.Bcerts file for the local system, but there will be no certificate.

User cannot get a local VeriSign certificate.

User can not get a certificate using ipsec_mgr. ipsec_mgr shows no certificate entry or an incomplete entry. The audit file shows the following error: Unable to obtain public/private key pair!

Check stdout for ipsec_mgr errors. Check the VeriSign OnSite Administrator’s Control Center for a pending request or existing certificate. Check the web proxy configuration (run ipsec_mgr, click on the Options menu, select System, then select Proxy Information). You may need to reset /var/adm/ipsec/cainfo.txt; remove /var/adm/ipsec/javabeans.txt, /var/adm/ipsec/certs.txt; revoke the old certificate, and restart the registration procedure.

If you cannot get a VeriSign certificate or if you forget to retrieve the certificate after you request it, ipsec_mgr may show a certificate entry for the local system. However, if you click Details the entry will be incomplete (there will be no information other than the IP address). When you attempt to initiate an ISAKMP/MM SA, it will fail and you will see errors similar to the following in the log file:

Msg: 201 From: IKMPD Lvl: ERROR Date: Mon Mar 11 16:19:39 2002
 Event: Unable to obtain public/private key pair!
Msg: 202 From: IKMPD Lvl: ERROR Date Mon Mar 11 16:19:39 2002
 Event: Process contruct error 0x1
Msg: 203 From IKMPD Lvl: ERROR Date Mon Mar 11 16:19:39 2002
 Event: Main Mode processing failed

If ipsec_mgr has problems retrieving a VeriSign certificate, it will write errors to the stdout device for ipsec_mgr (the device from which ipsec_mgr was started). The ipsec_mgr does not log errors using the HP-UX IPSec daemon.

If you request a VeriSign certificate using ipsec_mgr and ipsec_mgr cannot retrieve the certificate (the Check on Request operation fails), it is possible that the OnSite Administrator has not yet approved your certificate request. Have the OnSite Administrator use the VeriSign Control Center to check for pending requests. If the OnSite Administrator did not receive a request for approval, verify communication with the VeriSign CA. If you are using a web proxy server to access the VeriSign CA, verify the proxy server configuration (run ipsec_mgr, click on the Options menu, select System, then select Proxy Information).

Have the OnSite Administrator use the View Certificates area of the Control Center to check for an existing certificate. If a certificate already exists for your system, the VeriSign CA rejects any requests for a new certificate.

Restarting Registration

If VeriSign registration fails, the associated files may be left in an unusable state. You must reset them before trying to reregister. To do this, change the pending field in /var/adm/ipsec/cainfo.txt to false and verify that the VeriSign domain name and CA address are correct, as follows:

begin verisign pending: false domain: name_of_domain caserver: VeriSign_CA_address

You must also delete the /var/adm/ipsec/javabeans.txt and /var/adm/ipsec/certs.txt files.

Check the OnSite Administrator’s Control Center for an already existing certificate request or certificate, and deny or revoke it before restarting the registration process.

ISAKMP/MM SA negotiation succeeded, ISAKMP/MM SA established, but IPSec/QM SA negotiation failed.

Output from ipsec_report -sad does not show IPSec/QM SAs and the audit log contains Quick Mode processing failed or QM negotiation timeout error messages.

Run ipsec_policy to determine the IPSec policy. Check the transform list and lifetimes. Check the audit file.

If the ISAKMP/MM SA negotiation succeeded but the IPSec/QM SA negotiation failed, you will probably not see any ISAKMP/MM SAs in the output of the ipsec_report -mad command. This is because the HP-UX IPSec IKE daemon tears down an ISAKMP/MM SA if an IPSec/QM SA negotiation fails. To be sure that the ISAKMP/MM negotiation succeeded and that IKE actually attempted to negotiate the IPSec/QM SA, look for Quick Mode processing failed or QM negotiation timeout error messages in the audit file. A QM negotiation timeout error usually indicates that the remote system did not agree with the IPSec/QM SA proposal and chose not to respond.

Check which IPSec policy is being used with the ipsec_policy command. Check the IPSec policy configurations for mismatches.