[c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers

[Dustin Schuemann]

We have about 200 sites connected to us via GRE tunnels over IPSEC over MPLS for primary connectivity, and GRE over IPSEC over the Internet for backup, and EIGRP routing handling the failover.

Most of them are 2811HSEC/K9's, and they're working great. We've recently discovered issues with a couple of clients. They run fine over their primary GRE over IPSEC connection, but when they failover to backup we're losing certain packets (details will follow).

What we found is that they're all on either 1941's or 2911's, and are running 15.0Mx IOS with advanced IP services.  The rest of our clients are on 12.4T train, and none of them have any problems. We suspect it is an issue with the 15.x IOS.

Specifically, we're seeing two packets consistently lost. The first is a TCP 'SYN-ACK' from a telnet server, and the second is a UDP SIP REGISTER OK message. Both packets are quite small (well under 500 bytes), so I don't suspect an MTU issue. Packet captures both show that they're being encrypted and sent by the head-end, but are lost before they reach the decrypted tunnel interface. So either they're being lost in the path across the Internet, or the decryption is failing.

We see larger packets get through just fine, and other connections work great. We've opened a ticket with TAC but so far they have no clue.

Since these routers can't be downgraded to 12.4, our current plans are to ship a 2811HSEC bundle with an identical configuration to these clients to see if we can verify that it's a 15.0 issue, but I'm curious if anybody's seen anything similar, or if somebody who's more familiar than I am with bug tracker can find anything.