Encapsulating Security Payload (ESP)

ESP Encryption

ESP takes the data carried by IP, such as a TCP packet, and encrypts it using an encryption algorithm and cryptographic key. The output is ciphertext that is difficult to decode without knowing the key. The receiving IPSec ESP entity uses an associated decryption algorithm and the same key to extract the original data.

Figure 1-4 Symmetric Key Cryptosystem

The cryptography used by ESP is referred to as symmetric key cryptography or shared key cryptography because the sender and receiver must use the same key. In addition, the key must only be known by the sender and receiver, so this class of cryptography is sometimes referred to as secret key cryptography.

HP-UX IPSec supports the following encryption algorithms for ESP:

AES128-CBC is the most secure form of encryption for HP-UX IPSec. AES128-CBC encryption throughput rates are comparable to or better than DES-CBC and 3DES-CBC. For more information about HP-UX IPSec performance, refer to the HP-UX IPSec Sizing and Performance document available at www.docs.hp.com.

DES-CBC has been cracked (data encoded by DES has been decoded by a third party).

For added security, use ESP with authentication, as described in “ESP with Authentication and Encryption”.

Transport and Tunnel Modes

The ESP header can be used in transport mode or tunnel mode.

Transport Mode

In transport mode, the original IP header is followed by the ESP header. Only the upper-layer (e.g., TCP, UDP, IGMP) is encrypted. The IP header is not encrypted.

IPv6

In IPv6 ESP transport mode, IPSec inserts the ESP header after the following headers and extensions:

• the basic IPv6 header

• hop-by-hop options

• any destination options needed to interpret the ESP header

• routing extensions

• fragment extensions

The items listed below follow the ESP header and are encrypted:

• any destination options needed only for the “final” destination and not needed to interpret the ESP header

• the IP data or payload (e.g., TCP or UDP packet)

Figure 1-5 ESP Encryption in Transport Mode

Tunnel Mode

In tunnel mode, IPSec encloses, or encapsulates, the original IP datagram, including the original IP header, within a second IP datagram. All of the original IP datagram, including the original header, is encrypted. If ESP is used in tunnel mode on gateways, the outer, unencrypted IP header will contain the IP addresses of the gateways, and the inner, encrypted IP header will contain the ultimate IP source and destination addresses. This prevents eavesdroppers from detecting or analyzing traffic between the ultimate source and destination addresses.

IPv6

In IPv6 ESP tunnel mode, the packet layout is the same as IPv4 ESP tunnel mode, except that the original and new (outer) IP headers may include header extensions.

Figure 1-6 ESP in Tunnel Mode

ESP with Authentication and Encryption

The ESP encryption algorithms by themselves not provide authentication or guarantee data integrity, so you should use ESP encryption with an authentication and data integrity service. There are two ways to do this:

Authenticated ESP

With authenticated ESP, IPSec encrypts the payload using one symmetric key, then calculates an authentication value for the encrypted data using a second symmetric key and the HMAC-SHA1 or HMAC-MD5 algorithm. The ESP authentication value is appended to the end of the packet. The recipient computes its own authentication value for the encrypted data using the second symmetric key and the same algorithm. The recipient compares the result with the transmitted authentication value. If the values match, the recipient then decrypts the encrypted portion of the packet with the first symmetric key and extracts the original data.

Figure 1-7 Authenticated ESP

IPv6

The packet layouts and procedures for authenticated ESP and nested ESP in AH are the same for IPv6, except that the IP headers may include header extensions.

Figure 1-8 Nested ESP in AH