Advanced Troubleshooting

This section provides an overview of IPSec/9000 operation and troubleshooting scenarios and hints.

Overview of IPSec/9000 Operation

This subsection provides an overview of IPSec/9000 operation. This information is useful to further troubleshoot IPSec/9000 and analyze the data reported by the IPSec/9000 troubleshooting tools.

Figure 2-2 Outbound Processing

Outbound Data

Kernel Policy Engine
IPSec/9000 first checks the kernel policy engine's cache for an existing decision on the action to take (secure, drop, or pass in clear text) for the packet based on the IP addresses, protocol and port numbers. If the action is secure (uses an Authentication Header, AH or uses an Encapsulating Security Payload, ESP), there may be a reference to an existing IPSec SA that can be used.
Policy Manager Daemon
If no match is found in the policy engine's cache, the Policy Manager daemon is queried for the policy and action (secure, drop, or pass in clear text) to take.
The Policy Manager checks the hashed IPSec policies first and tries to find the policy with the IP packet filter that best matches the packet. The best match is the filter that matches the packet and has the largest number of hashable fields that match the packet. If multiple filters have the same number of hashable fields that match the packet, IPSec/9000 selects the first of these policies according to the order in the configuration file.
If IPSec/9000 finds no match in the hashed IPSec policies, it then sequentially searches the ordered IPSec policies for the first policy with an IP packet filter that matches the packet. If no match is found in the ordered policies, IPSec/9000 uses the default IPSec Policy.
If the transform (action) specified in the matching IPSec policy is to encrypt or authenticate the IP packets using AH or ESP, IPSec Security Associations may already exist for the policy. The new packet can use the existing IPSec SAs if the IP addresses, ports and protocols match. The new packet can also use the existing IPSec SAs if both IP addresses match and host-based keying is enabled (the Exclusive bit is not set for the policy). Otherwise, new IPSec SAs are established.
Establish an ISAKMP SA
If the transform (action) specified in the matching IPSec policy is to encrypt or authenticate the IP packets using AH or ESP and no IPSec Security Associations already exist, the IKE daemon (ikpmd) must establish an IPSec SA. Before establishing an IPSec SA, the IKE daemon must establish an ISAKMP SA with the remote system if none exists.
For the ISAKMP SA to be successfully established, both systems must authenticate the identity of the other system (primary authentication). This is done using pre-shared keys, or a method using security certificates and public-key encryption: RSA signatures.
In addition, both systems must use the same Oakley group (Diffie-Hellman initial values), and agree on the authentication algorithm and encryption algorithm used for the ISAKMP SA. They must also negotiate an SA lifetime.
Establish IPSec SAs
Once the ISAKMP SA is established, the IKE daemon uses the secured channel to establish IPSec SAs with its peers. Two IPSec SAs are established: one for packets from the local system to the remote system, and one for packets from the remote system to the local system.
For the IPSec SAs to be successfully established, both systems must agree on the type of transform (AH, ESP), including the authentication or encryption algorithm used. They must also negotiate SA lifetimes.
Add IPSec SAs to Kernel SA Database
The IPSec SAs are added to the kernel SA database by the IKE daemon. Each SA includes an SPI (Security Parameters Index) a number assigned by the receiving system to reference the SA. The SPI for outbound data is assigned by the remote system, while the SPI for the inbound data is established by the local system. The SPI is included in the AH or ESP header so that the destination system can process inbound packets with the correct SA parameters including encryption key(s).

Inbound Data

AH or ESP Packet
If the inbound packet has an Authentication Header (AH) and/or an Encapsulating Security Payload (ESP), IPSec/9000 checks the kernel SA database for inbound packets for an entry with the same SPI (Security Parameters Index) and source IP address. If one exists, it uses the information in the SA to properly decrypt or authenticate the packet. If the inbound packet has multiple SPIs (multiple transforms), IPSec/9000 searches the kernel SA database for each SPI.
If no matching entry exists, this is an error and IPSec/9000 sends an audit message to the audit daemon. IPSec/9000 discards the packet.
Clear Text Packet
If the inbound packet has no AH or ESP (it is a normal IP packet in clear text), IPSec/9000 must still determine whether the packet should be dropped or passed in clear text. IPSec/9000 checks the kernel policy engine's cache for an existing decision on the action to take (drop or pass in clear text) for the packet based on the IP addresses, protocol and port numbers. If the action is to apply an AH or ESP transform, IPSec/9000 sends an audit message to the audit daemon. This is because the remote system should have established IPSec SAs before sending the packet.
If no cache entry exists, IPSec/9000 queries the policy manager daemon for the appropriate action according to the IPSec policy with the filter that best matches the packet (or the default policy, if no filters match). Again, if the action is to apply an AH or ESP transform, IPSec/9000 discards the packet and sends an audit message to the audit daemon.

Troubleshooting Scenarios

Common troubleshooting scenarios, including their symptoms and resolutions, are described below.

Autoboot is Not Working Properly

Symptoms:	Autoboot fails
Action:	1 Set the IPSec/9000 password
	2 Set the Bootup option in ipsec_mgr
	3 Check that your policy is valid
	4 Execute: ipsec_admin -start
	5 Reboot the system again

If you are still having problems, contact your HP representative.

If IPSec/9000 is not using the IPSec policy that you expected, check for errors in the configuration file, such is incorrect IP addresses. Check the order of the IPSec policies--the order affects how IPSec/9000 selects policies (refer to "Configuration Reference, Selecting Policies" in chapter 1, Installing and Configuring IPSec/9000, for more information).

IPSec/9000 Incorrectly Passing Packets

Symptoms:	None, but no SAs will be established.
Action:	ipsec_report -sad (check for IPSec SAs)
	ipsec_report -policy (check for IPSec SAs)
	ipsec_ policy (determine policy)
	Check configuration file

If IPSec/9000 is mis-configured to pass packets that it should authenticate or encrypt, there will be no obvious external symptoms. Check that IPSec/9000 actually established Security Associations (SAs) and is encrypting/authenticating what you want. Check for IPSec/9000 SAs using the following command:

ipsec_report -sad

If there are no SAs for the IP packets that you expect and no user error, IPSec/9000 is probably mis-configured and passing packets it should not. Again, verify which IPSec policy is being used with the ipsec_policy command.

IPSec/9000 Incorrectly Attempting to Encrypt/Authenticate

Symptoms:	Link errors (unable to connect) on traffic that should not be encrypted/authenticated
Action:	ping, linkloop (check connectivity)
	ipsec_policy (determine policy)
	Check configuration file

If IPSec/9000 is mis-configured to encrypt and/or authenticate packets that it should not and the peer system is not configured to use IPSec/9000 encryption/authentication, you will consistently get connection errors (unable to connect or connection timed out). Again, verify which IPSec policy is being used with the ipsec_policy command.

IPSec/9000 Attempting to Encrypt/Authenticate and Failing

Symptoms:	Link errors (unable to connect)
Action:	ipsec_report -sad (check for IPSec SAs)
	ipsec_report -mad (check for ISAKMP SA)

If IPSec/9000 is configured to encrypt/authenticate but failing, it will appear as a connection error (unable to connect or connection timed out) to the user.

If users are consistently getting connection errors for traffic that should use IPSec/9000 for encryption or authentication, check for IPSec/9000 SAs using the following command:

ipsec_report -sad

Check for ISAKMP SAs using the following command:

ipsec_report -mad

No ISAKMP SA

Symptoms:	ipsec_report -mad output doesn't show SA
	Max Quick Modes > 1
Action:	ipsec_policy (determine ISAKMP policy)
	Check ISAKMP policy parms with remote: Oakley Group, Hash, Encryption, and (Primary) Authentication
	ipsec_admin -auditlv error
	ipsec_report -audit audit_file
	ipsec_admin -traceon, check UDP port 500

If PFS is not enabled and there is no ISAKMP SA to the remote, the ISAKMP SA negotiation is failing.

Check the ISAKMP policy with the remote configuration. Check that the Oakley Group (Diffie-Hellman group), hash, encryption and authentication fields match or are acceptable by the remote.

Increase the log level to "error."

ipsec_admin -auditlvl error

If, for example, you receive this audit message, "MM negotiation timeout," there may be a connectivity problem with the remote. Also, if the remote system is the responder, the remote system may not respond to the local system's proposals and cause a timeout.

Re-attempt to form the IPSec SAs by generating the appropriate network traffic and check the audit file:

ipsec_report -audit audit_file_name

(To determine the name of the current audit file, use the command: ipsec_admin -status.)

Use ipsec _admin -traceon udp to enable UDP tracing and verify that the packets are being sent and received by the correct system. ISAKMP always uses UDP port 500 to receive and send ISAKMP packets.

VeriSign Certificate-based ISAKMP Primary Authentication Fails

Symptoms:	ipsec_report -mad output doesn't show ISAKMP SA
Action:	Check for certificates in
	/var/adm/ipsec/certs.txt
	Check audit file for expired certificate, revoked certificate, certificate encoding problems.
	Try using preshared key authentication.

If you are using certificate-based primary authentication (RSA signature for ISAKMP primary authentication) check that you have a certificate for the local system in /var/adm/ipsec/certs.text. This file is encoded using IS0 ASN.1 (Abstract Syntax Notation 1) and is best viewed using ipsec_mgr.

Check the audit file to see if the IKE daemon is reporting any errors. The remote system's certificate may have expired or be on the Certificate Revocation List (CRL).

If you receive this audit message, "No local certificate available," you may need to use ipsec_mgr to be sure that you have loaded a certificate for the local system.

Try using pre-shared keys for primary authentication to see if that works. You will need to configure the same pre-shared key on both systems.

Entrust Certificate-based ISAKMP Primary Authentication Fails

Symptoms:	ipsec_report -mad output doesn't show ISAKMP SA
Action:	Check for Entrust profile (epf) in
	/var/adm/ipsec/username.epf
	Check audit file for expired certificate, revoked certificate, certificate encoding problems.
	Try using preshared key authentication.
	Contact your Entrust administrator

Make sure that Directory Services and the Entrust/Manager, listed in the entrust.ini file, can be reached by the local system.

Make sure that there are no policy rules interfering with the traffic flow between the HP 9000 system and Directory Services and also between the HP 9000 system and the Entrust Manager.

Check the IPSec/9000 GUI to be sure that the CA has been set to Entrust.

Check the audit file to see if the IKE daemon is reporting any errors. The remote system's certificate may have expired or be on the Certificate Revocation List (CRL).

Try using pre-shared keys for primary authentication to see if that works. You will need to configure the same pre-shared key on both systems.

Contact your Entrust administrator. The Entrust administrator should check that the necessary Entrust components are running.

Check that you have the Entrust Engine Library (libEntrust) installed on the local system. By default IPSec/9000 searches for this file under /usr/local/lib and SHLIB_PATH. Refer in Step 5A in chapter one for more Entrust information.

Pre-shared Key ISAKMP Authentication Fails

Symptoms:	ipsec_report -mad output doesn't show SA
Action:	Verify that key values match
	Check format (ASCII)
	Check audit file

If you are using pre-shared keys for primary authentication, check that the pre-shared key value is the same on both systems. IPSec/9000 uses ASCII while other vendors may use hexadecimal.

If you receive this audit message, "No preshared key for remote host," you may need to use ipsec_mgr to be sure that a pre-shared key is configured for the remote.

ISAKMP SA, No IPSec SAs

Symptoms:	ipsec_report -mad shows SA, no IPSec SAs in ipsec_sad
Action:	ipsec_policy (determine ISAKMP IPSec policy)
	Check transform list, lifetimes
	Check audit file

When responding, the non-HP-UX system must have lifetime values that fall within the HP-UX IPSec/9000 maximum and minimum ranges for the both the IPSec SA and the ISAKMP SA.

If the ISAKMP SA is being established with the remote system but no IPSec SAs are created, check that the transform list contains a transform that is acceptable by the remote system.

Check the lifetimes for the transform(s) and check that they are acceptable by the remote system.

Check the audit file for errors logged by the IKE daemon.

To verify if an ISAKMP SA is being established, increase the log level to "error."

ipsec_admin -auditlvl error

Generate traffic to form an IPSec SA and check the audit file: