[c-nsp] NAT "gaps" - packets not getting translated

[Ronan Mullally]

I've been struggling to get my head around this for the past few days
(trying to figure it out on a live box doesn't help).  I suspect I'm
missing something subtle (hopefully not obvious!).

I've got a setup like this:


         <---- VRF 1 ---->
   enduser - router --- router ---+
                                  |     A      B
                                  +-----+ 2811 +------ Internet
                                  +-----+
                                  |     A
   enduser - router --- router ---+
         <---- VRF n ---->

Packets at points A are typically 10.x.y.z, and can (but don't yet)
overlap in different VRFs.

These packets are NATed by the 2811 and have addresses of the form A.B.C.x
at point B.  Each VRF has a single IP address dedicated to it (so VRF 1
would be A.B.C.1, VRF 2 A.B.C.2, etc).

I have the following config on interfaces at A and B:

(A)
interface Port-channel1.1107
 description VRF 2
 mtu 1600
 encapsulation dot1Q 1107
 ip vrf forwarding VRF 2
 ip address 10.35.255.195 255.255.255.240
 ip mtu 1500
 ip nat inside
 ip virtual-reassembly
 standby 2 ip 10.35.255.193
 standby 2 follow vpn-vip
 standby 2 name vrf2-default
 crypto map CPE-vrf2 redundancy vrf2-default
end

(B)
interface Port-channel1.1011
 description Onwards Internet access via Sonicwall
 mtu 1600
 encapsulation dot1Q 1011
 ip address x.x.x.83 255.255.255.248
 ip mtu 1500
 ip nat outside
 ip virtual-reassembly
 standby delay minimum 30 reload 60
 standby 1 ip x.x.x.81
 standby 1 follow vpn-vip
 standby 1 authentication md5 xxxxxxxxxxxxxxxxx
 standby 1 name public-nat

The loopback interface on the 2811 for each VRF looks like:

interface Loopback1107
 description VRF 2 PWAN
 ip vrf forwarding VRF2
 ip address 10.35.255.252 255.255.255.255
 ip nat inside
 ip virtual-reassembly

I'm running BGP within each VRF to distribute routes.

I have the NAT mapping configured with:

 ip nat pool vrf1 A.B.C.1 A.B.C.1 netmask 255.255.255.128
 ip nat pool vrf2 A.B.C.2 A.B.C.2 netmask 255.255.255.128
 ip nat inside source list vrf1 pool vrf1 mapping-id 10 vrf VRF1 overload
 ip nat inside source list vrf2 pool vrf2 mapping-id 10 vrf VRF2 overload

(I've tried route-maps instead of source lists but it's made no
difference)

Most traffic works fine, however I'm seeing a steady stream of leaked
packets with internal source addresses, often ICMP traffic from loopback
interfaces on routers downstream within the various VRFs:

 10.0.0.49.1822 > x.x.x.x.21: S 1651477420:1651477420(0) win 65535 <mss 1360,nop,nop,sackOK>
 10.0.0.4 > x.x.x.x.49: ICMP time exceeded in-transit, length 36
 10.0.0.49.1822 > x.x.x.x.21: . ack 3704911765 win 65535

 (I can reproduce this at will using FTP connections)

 16:05:00.338256 IP 10.35.255.0.49154 > x.x.x.130.53: 55382+ A? <hostname>. (35)
 16:05:01.273139 IP 10.35.255.10.49154 > x.x.x.130.53: 19873+ A? <hostname>. (37)
 16:05:02.415973 IP 10.35.255.201 > x.x.x.130: ICMP time exceeded in-transit, length 36
 16:05:02.946458 IP 10.35.255.201 > x.x.x.130: ICMP time exceeded in-transit, length 36
 16:05:16.299198 IP 10.35.255.12.49154 > x.x.x.53: 19709+ A? <hostname>. (40)
 16:05:17.038419 IP 10.35.255.10.49154 > x.x.x.53: 35638+ A? <hostname>. (37)
 16:05:17.295820 IP 10.35.255.12.49154 > x.x.x.53: 19709+ A? <hostname>. (40)

 (it also happens regularly, but not always for DNS lookups)

After several days of head scratching I'm at a loss.  Does anybody have
any ideas?


-Ronan