Understanding Internet Protocol Security (IPsec)

IPSEC is the most popular form of VPN used today. It is important to understand how IPSEC works in order to trouble shoot issues with IPSEC tunnels. IPSEC is an end-to-end security scheme. This means that data is encrypted on one end and decrypted on the other end of the connection.

IPsec uses ESP or AH for initial encryption
ESP is the most popular method and is used in Barracuda VPNs

ESP Encapsulating Security Payload or ESP

In tunnel mode ESP adds a new ip header, after encrypting the original ip header and payload.

In transport mode ESP does not create a new header. It simply encypts the payload

In both modes an ESP header field is added to the packet.

ESP divides the date into smaller pieces in which it encrypts it with AES

Phase one: IKE1 (internet key exchange)

ISAKMP SA (phase1)

Occurs on UDP port 500
The server and client negotiate an encryption algorithm that will be used to transport the encryption keys to be used during the transfer of data.

This phase requires the following:

• • An encryption algorithm- This determines the depth and type of encryption.  (Keep in mind the deeper the encryption the slower the connection)
    • • AES (advanced encryption Standard).  The key size used for an AES cipher specifies the number of repetitions of transformation rounds that convert the input, called the plaintext, into the final output, called the ciphertext. The number of cycles of repetition are as follows:
        • • 10 cycles of repetition for 128-bit keys.
        • • 12 cycles of repetition for 192-bit keys.
        • • 14 cycles of repetition for 256-bit keys.
    • • 3DES – (Triple DES) they tripled the depth of the encryption of DES
    • • DES – A 56 bit algorithm that is susceptible to brute force attacks
    • • AES256 – Considered the top encryption cipher has the most combinations of possible keys
    • • Blowfish – A symmetric-key block cipher. This a good encryption cipher that encrypts in blocks
    • • Cast – another symmetric-key block cipher

• • A hashing algorithm
    • • SHA– (secure Hash Algorithm) this is a 160-bit hash
    • • MD5 – (message-digest algorithm) a 128-bit hash

• • Diffie-Hellman key (DH key ) also known as MODP Group on other Site 2 Site VPNs
    • • Group1 – 768-bit
    • • Group2 – 1024-bit
    • • Group5 – 1536-bit

• • Lifetime – Rekeying time in milliseconds.

Phase2: (Ike2) internet key exchange

 ISAKMP(phase2)

Phase two also  occurs over UDP port 500.

Phase two requires the same information that phase one needs in set up and also provides with one more layer of security through the form of authentication

Authentication can take place through the following means:

• • PSK (pre shared key) most commonly used, this is simply a pass phase that is shared.

• Client certificate

After the two IKE phases have completed data is transported through the establish VPN tunnel
The best way to troubleshoot IPSEC is to look at a packet capture. The IPSEC exchange is easy to see and identify in a packet capture.

Here is a sample packet capture showing the ISAKMP information you will need when troubleshooting both ends of a site 2 site VPN tunnel.

The packet you want look at is “Identity protection” this packet shows all of the phase one settings you are using to negotiate phase one between the two ends on the tunnel. There are several packets of this type. You will need to look at each until you find the one with the IKE attributes.

After opening the packet you will want to expand out the payload so you can see how each end is set up. You can also expand each IKE attribute to see what their individual settings are.
In this packet the initiating server is proposing the listed protocols it will be followed by an informational packet from the receiving server accepting the protocols or not accepting.

In the following packet the proposal was not chosen. This means that the IKE 1 settings are not the same on both devices. And the receiving device is rejecting the proposed IKE 1 settings.

Understanding How IPsec works will help you to set up and troubleshoot your Site to Site and client to site connections.