[c-nsp] Source-IP based NAT pickiness.
Andy Davidson
andy at nosignal.org
Thu Feb 9 10:20:52 EST 2006
Hi,
I would be pleased if someone could shed light on a problem we are seeing :
Essentially, there's a router at one of our offices with three
interfaces, one DMZ (GbitEthernet), one ATM where a BT Leased line
terminates, and one internal/office (also GbitEthernet), NATted network.
Everything is working fine, however, three devices on the Internal/NAT
network cannot reach the outside world (anywhere that their traffic
needs to be NATted). If their source-ip address changes, they can see
the outside world just fine. If we put a working device on the
source-ip that was affected, then this other device can't see out. More
below :
Symptoms :
----------
Some devices on our internal 10.22.10.0/23 subnet are capable of
reaching all routed and local networks (e.g. other subnets in the same
building, and networks in other buildings which are reached using
software tunnels), but not the internet. Most devices in this internal
network are capable of reaching internal and public resources.
It seems to be no co-incidence that the affected devices cannot reach
resources where the source-address is re-written with NAT. When a ping
is run from an affected device the 'sh ip nat statistics' value named
'Misses' increments.
Case Study :
------------
When a device has the address 10.22.10.83, it is able to reach all
internal resources, but no external resource.
A ping from this device to 85.234.132.110 is started. Initially we see
that no replies appear in the xterm.
Running tcpdump produces this output :
root at factory:~# tcpdump -i en1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes
16:48:53.151333 IP 10.22.10.83 > elvis.nosignal.org: icmp 64: echo
request seq 0
16:48:54.151559 IP 10.22.10.83 > elvis.nosignal.org: icmp 64: echo
request seq 1
16:48:55.151753 IP 10.22.10.83 > elvis.nosignal.org: icmp 64: echo
request seq 2
16:48:56.151947 IP 10.22.10.83 > elvis.nosignal.org: icmp 64: echo
request seq 3
On the router, we see :
howden-c2821-atm#sh ip nat translations
Pro Inside global Inside local Outside local Outside global
[...]
icmperr 194.73.16.246 10.22.10.83 --- --
howden-c2821-atm#sh ip nat statistics
Total active translations: 412 (0 static, 412 dynamic; 411 extended)
Outside interfaces:
ATM1/0.1
Inside interfaces:
GigabitEthernet0/1
Hits: 144280402 Misses: 17083116
CEF Translated packets: 155985395, CEF Punted packets: 8926797
Expired translations: 19205931
Dynamic mappings:
-- Inside Source
[Id: 1] access-list 2 interface ATM1/0.1 refcount 413
Queued Packets: 0
'show ip arp 10.22.10.83' indicates that the router is seeing traffic
from this computer, and it is the expected computer, meaning this is not
caused by an ip confict.
To spot whether the icmp request is getting as far as the router we try
the following on the router :
howden-c2821-atm#conf t
howden-c2821-atm(config)#logging trap debugging
howden-c2821-atm(config)#logging facility local2
howden-c2821-atm(config)#logging 10.22.0.161
howden-c2821-atm# debug ip nat detailed
This spools details of every packet which is natted to a loghost. We can
see whether our outbound packets are being natted with :
oddjob:~# tail -f /var/log/syslog | grep 10.22.10.83
Feb 6 16:55:32 10.22.0.1 774893: 775047: Feb 6 16:55:30.581 UTC: NAT*:
i: icmp (10.22.10.83, 1443) -> (85.234.132.110, 1443) [39494]
Feb 6 16:55:32 10.22.0.1 774894: 775048: Feb 6 16:55:30.581 UTC: NAT*:
i: icmp (10.22.10.83, 1443) -> (85.234.132.110, 1443) [39494]
Feb 6 16:55:32 10.22.0.1 774895: 775049: Feb 6 16:55:30.581 UTC: NAT*:
s=10.22.10.83->194.73.16.246, d=85.234.132.110 [39494]
Feb 6 16:55:33 10.22.0.1 774908: 775062: Feb 6 16:55:31.582 UTC: NAT*:
i: icmp (10.22.10.83, 1443) -> (85.234.132.110, 1443) [39498]
Feb 6 16:55:33 10.22.0.1 774909: 775063: Feb 6 16:55:31.582 UTC: NAT*:
i: icmp (10.22.10.83, 1443) -> (85.234.132.110, 1443) [39498]
Feb 6 16:55:33 10.22.0.1 774910: 775064: Feb 6 16:55:31.582 UTC: NAT*:
s=10.22.10.83->194.73.16.246, d=85.234.132.110 [39498]
Feb 6 16:55:34 10.22.0.1 774944: 775098: Feb 6 16:55:32.582 UTC: NAT*:
i: icmp (10.22.10.83, 1443) -> (85.234.132.110, 1443) [39524]
Feb 6 16:55:34 10.22.0.1 774945: 775099: Feb 6 16:55:32.582 UTC: NAT*:
i: icmp (10.22.10.83, 1443) -> (85.234.132.110, 1443) [39524]
Feb 6 16:55:34 10.22.0.1 774946: 775100: Feb 6 16:55:32.582 UTC: NAT*:
s=10.22.10.83->194.73.16.246, d=85.234.132.110 [39524]
Gosh - where's the inbound packets .. are our requests reaching the
target/destination device ?
elvis:~# tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
16:54:07.353123 IP 194.73.16.246 > elvis.nosignal.org: icmp 64: echo
request seq 0
16:54:07.353138 IP elvis.nosignal.org > 194.73.16.246: icmp 64: echo
reply seq 0
16:54:08.352888 IP 194.73.16.246 > elvis.nosignal.org: icmp 64: echo
request seq 1
16:54:08.352903 IP elvis.nosignal.org > 194.73.16.246: icmp 64: echo
reply seq 1
16:54:09.352635 IP 194.73.16.246 > elvis.nosignal.org: icmp 64: echo
request seq 2
16:54:09.352650 IP elvis.nosignal.org > 194.73.16.246: icmp 64: echo
reply seq 2
.. yes ! And the destination device is replying.
However, the inbound packets do not appear on the router's 'ip nat
detailed' debug log.
Notes :
-------
This problem is not just affecting traffic between this host and this
target - all traffic from 10.22.10.83 is affected, to all destinations.
Also affected is 10.22.10.61, and 10.22.10.66. There may be more addresses.
This has affected devices running Windows 2000, Windows XP, Linux, Mac OS/X.
It's not an MTU issue; this is set to 1500, and ping/tcp session
establishment data is tiny, so shouldn't be affected by a large mtu.
If we change the IP address on an affected device to an alternative one,
we do see the return packets in debug ip nat detailed :
oddjob:/var/log/HOSTS# tail -f 10.22.0.1/local2.log | grep 10.22.2.3
Feb 8 20:10:28 10.22.0.1 13970778: 13970584: Feb 8 20:10:25.112 UTC:
NAT: Allocated Port for 10.22.2.3 -> 194.73.16.246: wanted 40969 got 40969
Feb 8 20:10:28 10.22.0.1 13970779: 13970585: Feb 8 20:10:25.112 UTC:
NAT*: i: icmp (10.22.2.3, 40969) -> (85.234.132.110, 40969) [0]
Feb 8 20:10:28 10.22.0.1 13970780: 13970586: Feb 8 20:10:25.112 UTC:
NAT*: i: icmp (10.22.2.3, 40969) -> (85.234.132.110, 40969) [0]
Feb 8 20:10:28 10.22.0.1 13970781: 13970587: Feb 8 20:10:25.112 UTC:
NAT*: s=10.22.2.3->194.73.16.246, d=85.234.132.110 [0]
Feb 8 20:10:28 10.22.0.1 13970783: 13970589: Feb 8 20:10:25.140 UTC:
NAT*: s=85.234.132.110, d=194.73.16.246->10.22.2.3 [36314]
Feb 8 20:10:29 10.22.0.1 13970784: 13970590: Feb 8 20:10:26.120 UTC:
NAT*: i: icmp (10.22.2.3, 40969) -> (85.234.132.110, 40969) [1]
Feb 8 20:10:29 10.22.0.1 13970785: 13970591: Feb 8 20:10:26.120 UTC:
NAT*: s=10.22.2.3->194.73.16.246, d=85.234.132.110 [1]
Feb 8 20:10:29 10.22.0.1 13970787: 13970593: Feb 8 20:10:26.144 UTC:
NAT*: s=85.234.132.110, d=194.73.16.246->10.22.2.3 [36315]
Feb 8 20:10:30 10.22.0.1 13970788: 13970594: Feb 8 20:10:27.124 UTC:
NAT*: i: icmp (10.22.2.3, 40969) -> (85.234.132.110, 40969) [2]
Feb 8 20:10:30 10.22.0.1 13970789: 13970595: Feb 8 20:10:27.124 UTC:
NAT*: s=10.22.2.3->194.73.16.246, d=85.234.132.110 [2]
Feb 8 20:10:30 10.22.0.1 13970791: 13970597: Feb 8 20:10:27.148 UTC:
NAT*: s=85.234.132.110, d=194.73.16.246->10.22.2.3 [36316]
Feb 8 20:10:31 10.22.0.1 13970792: 13970598: Feb 8 20:10:28.728 UTC:
NAT: expiring 194.73.16.246 (10.22.2.3) icmp 45832 (45832)
we are running :
Cisco IOS Software, 2800 Software (C2800NM-ENTSERVICESK9-M), Version
12.4(1a), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Fri 27-May-05 21:02 by hqluong
ROM: System Bootstrap, Version 12.3(8r)T7, RELEASE SOFTWARE (fc1)
ROM: Cisco IOS Software, 2800 Software (C2800NM-IPBASE-M), Version
12.3(8)T8, RELEASE SOFTWARE (fc1)
More information about the cisco-nsp
mailing list