[c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers

[Vinny_Abello at Dell.com]

We've seen a couple of weird problems with 1921's running 15.0M(x)...

We've observed certain things like IPSec client functionality breaking when failing over to backup circuits which worked perfectly fine under older code and older routers that could run this code with the same configuration. The only workaround TAC could offer was "disable CEF"... of course definitely not ideal, but even more odd I cannot find the performance impact on the 1900 series ISR's with CEF disabled. The routing performance document from Cisco doesn't list anything in the column for the 1941 in the process switching columns... only the Fast/CEF switching. We haven't seen any performance issues in our customer environments where we have to do this to fix functionality, but I'd much appreciate it if CEF actually worked with the feature sets in the router. Another thing that doesn't work with CEF enabled in this code train is terminating an IPSec tunnel on a loopback interface. Works ok in other version and works fine if I disable CEF.

-Vinny

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Dustin Schuemann
Sent: Sunday, September 25, 2011 6:01 PM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] GRE over IPSEC loss in IOS 15.x / ISR x9xx Routers

We have about 200 sites connected to us via GRE tunnels over IPSEC over MPLS for primary connectivity, and GRE over IPSEC over the Internet for backup, and EIGRP routing handling the failover.

Most of them are 2811HSEC/K9's, and they're working great. We've recently discovered issues with a couple of clients. They run fine over their primary GRE over IPSEC connection, but when they failover to backup we're losing certain packets (details will follow).

What we found is that they're all on either 1941's or 2911's, and are running 15.0Mx IOS with advanced IP services.  The rest of our clients are on 12.4T train, and none of them have any problems. We suspect it is an issue with the 15.x IOS.

Specifically, we're seeing two packets consistently lost. The first is a TCP 'SYN-ACK' from a telnet server, and the second is a UDP SIP REGISTER OK message. Both packets are quite small (well under 500 bytes), so I don't suspect an MTU issue. Packet captures both show that they're being encrypted and sent by the head-end, but are lost before they reach the decrypted tunnel interface. So either they're being lost in the path across the Internet, or the decryption is failing.

We see larger packets get through just fine, and other connections work great. We've opened a ticket with TAC but so far they have no clue.

Since these routers can't be downgraded to 12.4, our current plans are to ship a 2811HSEC bundle with an identical configuration to these clients to see if we can verify that it's a 15.0 issue, but I'm curious if anybody's seen anything similar, or if somebody who's more familiar than I am with bug tracker can find anything.
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/