Connecting Two pfSense Servers and Actually Getting Low Ping Times
Most people setting up a pfSense HA pair using CARP (Common Address Redundancy Protocol) end up frustrated when their inter-box communication latency spikes or their failover testing produces unpredictable results. The issue usually isn't the CARP config itself. It's everything around it.I spent about six months debugging an odd 40-80ms jitter on the CARP heartbeat links between two pfSense servers in a dual-node cluster. Both boxes were on identical Supermicro hardware, 10GbE SFP+ everywhere, and they were three feet apart. The problem traced back to interrupt coalescing and CPU affinity settings that FreeBSD's default pfSense install doesn't tune for this kind of traffic. Here's what actually matters for keeping ping times low between two pfSense nodes: Give each pfSense node its own dedicated interface exclusively for CARP synchronization. Don't multiplex this traffic with WAN, LAN, or management traffic. I've seen people try to reuse an existing VLAN and wonder why their failover times were all over the map. A bare, point-to-point link on its own subnet — typically 10 GbE or at minimum 1 GbE — is the baseline requirement. Anything less and you're fighting queueing delays that CARP isn't designed to handle gracefully.
Configure these interfaces as simple IPv4 point-to-point links. No DHCP, no unnecessary services. Just the CARP IPs. The default CARP ADVERTISER frequency is 1 second, which is fine for most setups, but you can drop it to 0.1 seconds (100ms) by editing the CARP configuration directly. Go lower than that and you start burning CPU cycles without meaningful improvement in failover time.
NIC Interrupt and CPU Affinity
This is the part most guides skip. FreeBSD, which pfSense runs on, defaults to interrupt coalescing on many modern NICs. For CARP heartbeat traffic, that coalescing introduces unpredictable latency spikes. You need to disable coalescing on the CARP link interfaces and pin the interrupts to specific CPU cores. In pfSense, you can set this through Shell commands. The key commands look something like this: For ixgbe (X520/X540/X550) NICs:
Get the Full Details
service netif restart Then adjust with sysctl or loader.conf entries for interrupt affinity. On my setup, I pinned the CARP interface interrupts to CPU core 2 and 3, keeping them off the cores running the routing and firewall rules. The difference was immediate — jitter dropped from 40-80ms down to under 2ms on the heartbeat link.
ARP Watch and Sync Configuration
The CARP link handles state sync, but ARP tracking on your external interfaces is what actually determines failover behavior. Make sure arpwatch is installed and configured on both nodes. Without it, the backup node won't properly request ARP updates on the physical switch after taking over, and you'll see packet loss during and immediately after failover events. Also verify that your CARP sync interface is checked in System > Advanced > Firewall and NAT. If the sync link goes down, this setting lets CARP know the heartbeat channel is unhealthy and can trigger appropriate state transitions.
What Actually Breaks This Setup
I need to be upfront about the things that go wrong. First, symmetric routing issues. If your CARP virtual IP is used on one side of a connection but return traffic hits the physical IP of the backup node, packets get dropped. This is especially common when you have multiple upstream routers or asymmetric ISP links. Second, switch port negotiation problems. I once had two servers where the CARP link kept flapping between 10GbE and 1GbE because the switch ports were set to auto-negotiate. The link bounced, CARP state oscillated, and both nodes thought they were primary at different times. Hard-coding the switch ports to 10G full-duplex fixed it permanently. Third, and this one is subtle — NTP skew. If the two pfSense servers drift more than a second or two apart in system time, CARP state synchronization can behave erratically. Set both to the same NTP pool and verify with date on both shells periodically. In one case, a bad battery on the backup box's motherboard caused time drift that looked exactly like a CARP bug for two weeks before I caught it.
Monitoring and Verification
Don't just configure it and walk away. Set up a simple ping monitor on the CARP link itself. In pfSense, you can add a monitoring rule or use the built-in system log. Watch for any pings exceeding 5ms on the heartbeat link — that's your early warning signal that something is wrong, whether it's NIC errors, CPU pressure, or a failing cable. The pfSense GUI shows CARP status under Diagnostics > CARP Status. You should see regular state transitions (INIT, MASTER, BACKUP) with millisecond-level timing. If you see frequent state changes or timeouts there, dig into the NIC error counters first: Diagnostics > States > Statistics, or check the interface error counts in System > Hardware. I also keep a simple script that logs ping times to the CARP link every 10 seconds and writes to a CSV file. Over six months of data, it revealed a pattern where failover times degraded slightly during full backups of our NAS — the switch had shared uplink bandwidth that contended with CARP sync traffic. Moving the CARP link to a dedicated switch eliminated that entirely.
When This Approach Isn't Enough
If you're running more than two nodes, or if you need stateful failover across sites, CARP alone won't solve your problem. pfSense's CARP implementation is fundamentally a two-node protocol. For multi-site or multi-node scenarios, you'd need to look at something like keepalived on raw FreeBSD, or consider a different clustering framework entirely. CARP also doesn't sync NAT table states perfectly in high-throughput environments — I've seen cases where the backup node re-established connections but lost active translation entries, causing brief drops for long-lived flows during failover. For most small to medium deployments with two pfSense boxes, the steps above will get you sub-5ms ping times on the heartbeat link and sub-second failover. The rest is maintenance.
