IPsec VPN, Main mode Vs Aggressive mode

In order to secure data transmission, IPsec VPN uses 2 IKE phases

1. IKE phase 1:  the goal of this phase is to establish an encrypted, authenticated channel so the peers involved in tunnel construction  can securely negotiate security parameters in order to secure data exchange. During this phase the peers will:
1. negotiate security parameters (encryption algorithms, hash function…) to be used to  secure IKE phase 2 exchanges
2. uses Deffie-Hellman to generate a shared key
3. Do mutual authentication. This authentication can use Pre-Shared Key (PSK), digital signature or  public key encryption.

This phase can be conducted in main mode or aggressive mode

2. IKE phase 2: security parameters to be used to secure data transmission will be negotiated

 So what is the difference between main and aggressive modes?

1. Case 1: Authentication with PSK

Main Mode

There is an exchange of 6 messages. The following figure show this exchange in case of PSK authentication being used

Where :

• HDR: is the ISAKMP header
• SA (Security Association): SA negotiation payload  (encryption algorithms, hash function, Deffie-Hellman group…) with one or more proposals. An Initiator may provide multiple proposals for negotiation; a Responder must reply with only one.
• KE (Key Exchange): a payload that contains public information exchanged in a Deffie-Hellman exchange in order to generate a DH shared key.
• Ni, Nr:  nonce payload for Initiator and Responder respectively, it is a random number meant to be used once. Nonce is used as an input in key generation algorithm and is also used as an anti-replay mechanism to prevent an attacker to replay an old message.      
• IDii, IDir: Identity payload for the Initiator and the Responder respectively. The identity can be IPv4 address, IPv6 address, FQDN…
• HASH_I, HASH_R: this is the hash payload for the Initiator and the Responder respectively. Pre-Shared key, IDii and IDir are among inputs used to produce the hash payload
• HDR*: denotes that ISAKMP payload is encrypted, this mean that identities (IDii and IDir) are protected during authentication exchanges (the last 2 messages)

The shared master secret SKEYID is generated before the third exchange (messages 5&6)

           SKEYID = prf(pre-shared-key, Ni_b |  Nr_b)

In this mode:                                                                                                  

• Identities and Hash are protected.
• PSK is used as in input to generate the shared master secret, as such the Responder must know in advance which PSK to associate to the Initiator.
  The shared secret is generated before the third exchange (messages 5&6), so the responder don’t know yet the initiator’s identity in order to make a correct PSK association, therefore only the Initiator’s IP address could be used to choose the right PSK, this address must, therefore, be configured on the Responder, consequently, Main mode with PSK doesn’t support dynamic addressing. 

Aggressive Mode

There is only an exchange of 3 messages. This is illustrated bellow in case of PSK authentication being used

As we can see, there is no payload encryption as in main mode, the identities (IDii and IDir) and hash payload which are used in nodes authentication are not protected, therefore:

·        An observer could determine the parties performing negotiation

·        It is possible to intercept the hash and to retrieve the PSK  by conducting a dictionary or brute force attack. A description of this attack can be found here 

·        Since  SA and KE are exchanged in the same time, the Diffie-Hellman group can’t be negotiated

The responder receives the initiator’s identity in the same time it receives  the Key Exchange materials, in this case the correct PSK association can be done even if the initiator is using dynamic IP address

2. Case 2: Authentication with Digital Signatures

• SIG_I, SIG_R: Hash signatures for the Initiator and the Responder respectively.
• [Cert]: certificates of the Initiator and the Responder

The shared master secret SKEYID is generated before the third exchange (messages 5&6)

           SKEYID = prf(Ni_b | Nr_b, g^xy)     ;  g^xy is the shared DH secret

Instead of including HASH_I and HASH_R as in PSK, the Hashes are signed and the signatures are sent instead.

As with PSK, identities are protected

The Shared master secret is generated using parameters exchanged during Key Exchange, so there is no information that the Responder must know in advance as with PSK, therefore the dynamic addressing is supported

Aggressive Mode

As with PSK

·        Identities are still not protected, but the authentication is much more secure as it is based on digital signature.

·        The DH can’t be negotiated

·        Dynamic IP addressing is supported

3. Case 3: Public Key Encryption  

• <IDix_b>PubKey_y : indicate that IDix_b is encrypted with the public key of "y"
• Hash(1): the hash of the certificate containg the responder’s public key  that  the initiator is using to encrypt the nonce and  the identity. This parameter is optional, and is sent in case the responder has several public keys.

 The shared master secret SKEYID is generated before the third exchange (messages 5&6)

           SKEYID = prf(hash(Ni_b | Nr_b), CKY-I |  CKY-R)

The identities are protected by public key encryption. The dynamic addressing mode is supported as the Shared master secret is generated using parameters exchanged during Key Exchange

Aggressive Mode

• Unlike  PSK and Digital Signature aggressive modes, the identities here are protected
• The DH can’t be negotiated
• Dynamic IP addressing is supported

4. Conclusions

• Aggressive mode is faster than main mode
• It is generally recommended to use main mode instead of aggressive mode.
• If aggressive mode must be used, for performance issue for example, prefer Public Key Encryption authentication.