Let's Get This Straight Before Anyone Starts Arguing
IPsec has two modes, ESP can run in either one, and people regularly deploy them wrong because they don't actually understand what the difference means for their traffic. I've spent more years than I care to count watching engineers tie themselves in knots over this. The configuration that works depends entirely on what problem you're trying to solve, not on whichever mode sounds more impressive in a whitepaper. Transport mode protects the payload of a single IP packet. The original IP header stays exactly where it is, untouched. ESP wraps around the upper-layer data and adds its own authentication trailer. The result is that the source and destination addresses in the outer header are the real endpoints, and whatever sits inside the ESP wrapper is encrypted and authenticated. This is the mode you use for host-to-host security, like when two servers need to talk and you want to protect their traffic without changing anything about the routing path. Tunnel mode wraps the entire original IP packet inside a new IP header. ESP sits between the outer header and the inner packet. The outer header carries the tunnel endpoints, which are usually your VPN gateways or routers. The inner header retains the real source and destination of the actual communication. This is the mode you deploy for site-to-site VPNs, or when you need to hide the real topology behind a gateway. The packet leaves one tunnel endpoint looking like normal IP traffic to the outside world, then arrives at the other endpoint, gets stripped, and continues on its way with the original addressing intact.
Here's the thing most people miss. Transport mode does not provide network-layer privacy for the packet header. If someone is sniffing traffic between two hosts using transport mode, they can still see the source IP, destination IP, and the DSCP/ECN bits. Everything after the ESP header is protected, but the routing information is visible. Tunnel mode hides the inner header from anyone watching the wire between the two tunnel endpoints. That's a fundamental difference, not a cosmetic one. I ran into a real problem last year involving a customer who had three sites connected via IPsec tunnel mode between their routers. They wanted to add host-to-host encryption between two specific servers at each site, so they configured ESP transport mode on those servers. It looked fine on paper. The issue was that the transport mode SA and the tunnel mode SA both matched the same traffic flow because the tunnel was terminating at the router, not the server. The routing table sent the traffic into the tunnel first, and the transport mode SA on the server never triggered. The servers appeared to be running transport mode encryption, but they were actually sending unencrypted traffic through the tunnel. I had to adjust the ACLs on the tunnel to exclude that specific host-to-host flow, forcing it to the transport SA instead. Took about twenty minutes once I figured out what was happening, but diagnosing it took longer than I'd like to admit.
How To Actually Configure Both Modes
The configuration steps vary depending on your platform, but the logic is identical everywhere. For Linux with strongSwan, which is what I usually work with, transport mode looks something like this in the ipsec.conf file. You define a connection with left and right set to the actual host addresses, mode set to transport, and the IKE proposal along with the ESP proposal configured with your preferred cipher and hash. IKEv1 or IKEv2 both work, though IKEv2 is the standard now and handles rekeys more cleanly. StrongSwan will negotiate the SA, and then traffic between those two hosts gets encrypted in transport mode automatically. For tunnel mode, you change the connection type and add subnet definitions for leftsubnet and rightsubnet. These define which traffic should be captured and tunneled. The left and right addresses become your tunnel endpoints, usually the public or uplink interfaces of your routers. The subnets are the private networks behind each endpoint. When traffic matches those subnets, strongSwan creates the tunnel SA and encapsulates the packets.
Get the Full Details
On Cisco IOS, the syntax is different but the concept is the same. You create a crypto isakmp policy, then a crypto ipsec transform set, then a crypto map. The transform set specifies esp-aes esp-sha-hmac or whatever algorithm combination you're using. The crypto map references the transform set and defines the peer and the ACL that matches the traffic to protect. For transport mode, the ACL matches the specific host traffic. For tunnel mode, the ACL matches the internal subnets. Aws and Azure have their own implementations, obviously, but the underlying mode distinction remains identical. Site-to-site VPNs in the cloud use tunnel mode by default. Point-to-site configurations can use either, though tunnel mode is more common because it gives you the gateway abstraction most people want.
What Nobody Tells You About Mode Selection
There are two counter-intuitive things about IPsec mode selection that trip people up regularly. First, transport mode does not require a VPN gateway. You can encrypt traffic directly between two endpoints without any intermediary device. This matters when you're dealing with environments where you can't install or configure a VPN concentrator, or when you need encryption between hosts that aren't behind a perimeter device. Many people assume IPsec always needs a gateway. It doesn't. Transport mode is purely endpoint-to-endpoint. Second, tunnel mode adds roughly forty to sixty bytes of overhead per packet depending on your algorithm choices and key lengths. ESP adds its own header and trailer, the new IP header adds twenty bytes, and you're also dealing with additional fragmentation risk if your MTU isn't adjusted. Most people deploy tunnel mode without lowering the MTU on the tunnel interface, and then they spend weeks troubleshooting intermittent connectivity issues that turn out to be fragmentation problems. TCP segmentation fails silently, UDP packets get dropped, and application performance degrades in ways that look like a network issue rather than a configuration issue. Set the MTU to 1360 or lower on tunnel interfaces, or enable PMTUD with IPv4, and save yourself the headache.
I once saw a production environment where a hospital's imaging system went intermittent after someone enabled a site-to-site IPsec tunnel. The tunnel was adding overhead that pushed some DICOM packets above the MTU, and the PMTUD path was blocked by a misconfigured ICMP setting somewhere in the chain. The imaging workstation would connect, then hang on large file transfers. We traced it to the tunnel MTU mismatch. Setting the interface MTU to 1350 fixed it immediately. The root cause wasn't the tunnel itself, it was the assumption that tunnel mode wouldn't affect packet sizing.
When Transport Mode Is The Wrong Tool
Transport mode fails when you need to route through an intermediary. If your traffic passes through a NAT device, transport mode usually breaks because the IP header is modified by the NAT but the ESP authentication covers that header. The receiver sees a modified header, the integrity check fails, and the packet is discarded. Tunnel mode handles NAT traversal because the outer header is what gets modified, and that header isn't part of the ESP authentication payload. NAT-T exists as a workaround for transport mode, but it's an additional complexity that introduces its own failure modes. If you're going through NAT, use tunnel mode unless you have a very good reason not to. Transport mode also doesn't help when you're trying to hide internal network topology. If your goal is security through obscurity at the network layer, transport mode exposes your addressing. Tunnel mode wraps everything inside the tunnel headers, which is why it's the standard for site-to-site deployments. The outer header only reveals the tunnel endpoints, not the actual communicating hosts or their subnets.
Authentication vs Encryption Tradeoffs
ESP can operate in three flavors. Encapsulating Security Payload with encryption only, with authentication only, or with both. The default and recommended setting is both. Authentication-only ESP exists but is rarely useful. Encryption-only ESP existed in early implementations and is deprecated in current standards because unauthenticated encryption is vulnerable to bit-flipping attacks and replay manipulation. Don't use encryption-only ESP. There's no legitimate reason for it in production. When you're choosing algorithms, AES-256 with SHA-256 or SHA-384 is the current standard for most deployments. AES-GCM is better if your hardware supports it because it provides authenticated encryption in a single operation, which reduces CPU overhead compared to separate encryption and authentication steps. Older hardware might only support AES-CBC with HMAC, which is fine but less efficient. Desperate situations sometimes use ChaCha20-Poly1305, which is software-friendly and fast on devices without AES-NI, but it's not universally supported across all IPsec implementations. I've seen teams configure ESP with 3DES because it was the default on an older device they were replacing. 3DES is deprecated in RFC 7565 and should not be used in any new deployment. It's slower than AES, provides weaker security margins, and creates compliance issues in environments that require FIPS 140-2 validation. Just don't do it.
Debugging Tips That Actually Work
When IPsec isn't working, most people check the wrong thing first. They look at the SA status and see established connections, assume everything is fine, and then wonder why traffic isn't flowing. The SAs might be established but the traffic selectors could be wrong. Always verify that the ACL or traffic selector matching the SA actually covers the traffic you expect to be encrypted. On Linux, the command is setkey -D to dump SAs and setkey -s to see policy information. The policy is what actually determines whether traffic gets encrypted or sent in the clear. An established SA without a matching policy won't encrypt anything. I've debugged this exact scenario three times in the past year alone. On Cisco, show crypto sa details and show crypto map give you the full picture. Look at the match address field in the crypto map and confirm it aligns with your traffic. The transform set should match between local and remote. Mismatched transform sets cause the most common negotiation failures, and they fail silently in the sense that the tunnel appears to be configured correctly but never actually establishes.
Packet capture with tcpdump or Wireshark filtered on esp or protocol 50 is the most reliable way to confirm whether traffic is actually being encrypted. If you send a test packet and the capture shows clear-text IP followed by ESP, the SA is working. If you see just regular IP with no ESP component, the policy isn't matching and you need to check your selectors, not your algorithms. UDP port 4500 appears in NAT-T scenarios. If you're seeing traffic on that port unexpectedly, check whether NAT traversal is enabled on both endpoints. Asymmetric NAT-T configuration, where one side has it enabled and the other doesn't, causes intermittent tunnel failures that are extremely difficult to diagnose because the tunnel appears to work most of the time.
The Real Answer To The Debate
Transport mode and tunnel mode aren't competing solutions. They solve different problems. Transport mode is for endpoint-to-endpoint encryption where you don't need or want a gateway. Tunnel mode is for gateway-mediated encryption where you need to protect entire subnets or hide topology. The debate exists because people try to use one mode for the other's job, and then they wonder why it doesn't work. If you're connecting two servers directly and both support IPsec natively, use transport mode. It's simpler, has less overhead, and doesn't require a gateway device. If you're connecting two networks through a perimeter firewall or VPN concentrator, use tunnel mode. It's the only mode that makes sense for that topology. Hybrid deployments exist and they're valid. A site-to-site tunnel mode VPN with transport mode extensions between specific hosts inside each site is a reasonable configuration when you need both network-level and host-level encryption. The caveat is that you need to be careful about SA selection order and ensure your policies don't conflict, as I described with the earlier example.
Performance differences between the modes are negligible on modern hardware. The real cost is in configuration complexity and operational overhead, not in throughput. Tunnel mode's additional encapsulation adds latency measured in microseconds on competent equipment, and the MTU adjustment is a one-time configuration task. Don't avoid tunnel mode because of performance concerns unless you're working with constrained embedded devices. Both modes support reverse path filtering gracefully when configured correctly. Some teams disable reverse path filtering on tunnel interfaces to avoid packet drops, which is a workaround for a configuration issue rather than a solution. Fix the rpf settings or use strict reverse path filtering on the correct interface, and you won't need the workaround. IPsec is straightforward when you understand what each mode actually does. It becomes a nightmare when you copy configuration from a template without verifying the mode matches your use case. Check your traffic selectors, verify your policies, test with a packet capture, and confirm the mode you chose actually matches the problem you're solving. That process takes about ten minutes and prevents most of the issues I've encountered over the years.
