[c-nsp] CBAC not properly handling fragmentation?
Marc Haber
mh+cisco-nsp at zugschlus.de
Sat Jan 28 07:20:43 EST 2006
Hi,
this is my inaugural posting to cisco-nsp ;)
I have worked with Cisco routers from 1999 through 2004, thus my
experience is a little bit outdated. Additional, I have absolutely no
experience with CBAC.
The issue is a Cisco 1803 with IOS 12.4(4)T1. As long as CBAC is
enabled, Downloads from the Internet (regardless of http, pop3 or ssh)
into the LAN stall after a few kilobytes (sometimes 30, sometimes 90)
have been transferred.
Here are the relevant parts of the configuration:
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec
service timestamps log datetime msec
service sequence-numbers
no service dhcp
ip subnet-zero
no ip source-route
ip cef
ip tcp synwait-time 10
ip inspect one-minute high 1100
ip inspect udp idle-time 60
ip inspect dns-timeout 10
ip inspect name DEFAULT101 icmp
ip inspect name DEFAULT101 netshow
ip inspect name DEFAULT101 cuseeme
ip inspect name DEFAULT101 streamworks
ip inspect name DEFAULT101 udp
ip inspect name DEFAULT101 tcp
ip inspect name DEFAULT101 ftp
ip inspect name DEFAULT101 h323
ip inspect name DEFAULT101 sqlnet
ip inspect name DEFAULT101 rcmd
ip inspect name DEFAULT101 realaudio
ip inspect name DEFAULT101 vdolive
ip inspect name DEFAULT101 tftp
ip inspect name DEFAULT101 rtsp
ip ips po max-events 100
controller DSL 0
mode atm
line-term cpe
line-mode 2-wire line-zero
dsl-mode shdsl symmetric annex B
line-rate auto
interface ATM0
no ip address
no ip redirects
no ip proxy-arp
ip route-cache flow
no atm ilmi-keepalive
interface ATM0.2 point-to-point
no ip redirects
no ip proxy-arp
pvc 1/32
pppoe-client dial-pool-number 1
interface Vlan1
description $ETH-SW-LAUNCH$$INTF-INFO-FE 1$$FW_INSIDE$
ip address 192.168.0.250 255.255.255.0
ip access-group 101 in
no ip redirects
no ip proxy-arp
ip nat inside
ip inspect DEFAULT101 in
ip virtual-reassembly
ip route-cache flow
ip tcp adjust-mss 1452
interface Dialer0
description $FW_OUTSIDE$
mtu 1456
ip address negotiated
ip access-group 104 in
no ip redirects
no ip proxy-arp
ip nat outside
ip virtual-reassembly
encapsulation ppp
ip route-cache flow
ip tcp adjust-mss 1452
dialer pool 1
dialer-group 1
no cdp enable
ppp authorization noauthen
ppp chap hostname <removed>
ppp chap password <removed>
ppp pap sent-username <removed> password <removed>
crypto map SDM_CMAP_1
ip classless
ip route 0.0.0.0 0.0.0.0 Dialer0
ip nat inside source route-map SDM_RMAP_1 interface Dialer0 overload
access-list 101 deny ip host 255.255.255.255 any log
access-list 101 deny ip 127.0.0.0 0.255.255.255 any log
access-list 101 permit ip any any
access-list 103 deny ip 192.168.0.0 0.0.0.255 192.168.124.128 0.0.0.15
access-list 103 deny ip any host 192.168.124.100
access-list 103 deny ip any host 192.168.124.101
access-list 103 deny ip any host 192.168.124.102
access-list 103 deny ip any host 192.168.124.103
access-list 103 deny ip any host 192.168.124.104
access-list 103 deny ip any host 192.168.124.105
access-list 103 deny ip any host 192.168.124.106
access-list 103 deny ip any host 192.168.124.107
access-list 103 deny ip any host 192.168.124.108
access-list 103 deny ip any host 192.168.124.109
access-list 103 deny ip any host 192.168.124.110
access-list 103 deny ip any host 192.168.124.111
access-list 103 deny ip any host 192.168.124.112
access-list 103 deny ip any host 192.168.124.113
access-list 103 deny ip any host 192.168.124.114
access-list 103 deny ip any host 192.168.124.115
access-list 103 deny ip any host 192.168.124.116
access-list 103 deny ip any host 192.168.124.117
access-list 103 deny ip any host 192.168.124.118
access-list 103 deny ip any host 192.168.124.119
access-list 103 deny ip any host 192.168.124.120
access-list 103 deny ip any host 192.168.124.121
access-list 103 deny ip any host 192.168.124.122
access-list 103 deny ip any host 192.168.124.123
access-list 103 deny ip any host 192.168.124.124
access-list 103 deny ip any host 192.168.124.125
access-list 103 deny ip any host 192.168.124.126
access-list 103 deny ip any host 192.168.124.127
access-list 103 deny ip any host 192.168.124.128
access-list 103 deny ip any host 192.168.124.129
access-list 103 deny ip any host 192.168.124.130
access-list 103 permit ip 192.168.0.0 0.0.0.255 any
access-list 104 permit ip 192.168.124.128 0.0.0.15 192.168.0.0 0.0.0.255
access-list 104 permit ip host <remote-management1> any
access-list 104 permit ip host <remote-management2> any
access-list 104 permit tcp any any eq domain
access-list 104 permit udp any any eq domain
access-list 104 permit ahp any any
access-list 104 permit esp any any
access-list 104 permit udp any any eq isakmp
access-list 104 permit udp any any eq non500-isakmp
access-list 104 permit icmp any any echo
access-list 104 permit icmp any any echo-reply
access-list 104 permit icmp any any time-exceeded
access-list 104 permit icmp any any unreachable
access-list 104 permit ip 192.168.0.0 0.0.255.255 any
access-list 104 deny ip 127.0.0.0 0.255.255.255 any log
access-list 104 deny ip host 255.255.255.255 any log
access-list 104 deny ip host 0.0.0.0 any log
access-list 104 deny ip any any log
route-map SDM_RMAP_1 permit 1
match ip address 103
I have stripped ipsec and isdn configuration, Fa0 and the switch
interfaces. If that is important, I'll happily deliver these parts of
configuration as well.
Connection to the Internet is "T-DSL Business Symmetrisch 2 Mbit/s"
which is the business SDSL connection offered by Deutsche Telekom, the
major carrier in Germany. Technically, it is PPPoE with one fixed IP
address, and the carrier information differs whether MTU is 1452 or
1456. The 1803 does souce NAT for the internal network which is on
RFC1918 site local addresses.
As soon as CBAC is active, packets coming in over the PPPoE session
arrive in the local LAN truncated.
For debugging, I did an http download from a Linux webserver under my
control on the Intenet to a Linux client in the LAN and tcpdumped on
both boxes. The problem can also be reproduced on the Windows-boxes in
the LAN.
Here the tcpdump, as the http server on the Internet sees it on its
ethernet interface a.b.c.d. e.f.g.h is the fixed IP address of the
Cisco 1803.
15:39:02.274022 IP e.f.g.h.45268 > a.b.c.d.80: tcp 0
15:39:02.274135 IP a.b.c.d.80 > e.f.g.h.45268: tcp 0
15:39:02.311319 IP e.f.g.h.45268 > a.b.c.d.80: tcp 0
15:39:02.328137 IP e.f.g.h.45268 > a.b.c.d.80: tcp 176
15:39:02.328207 IP a.b.c.d.80 > e.f.g.h.45268: tcp 0
15:39:02.330622 IP a.b.c.d.80 > e.f.g.h.45268: tcp 1440
15:39:02.330695 IP a.b.c.d.80 > e.f.g.h.45268: tcp 1440
15:39:02.330734 IP a.b.c.d.80 > e.f.g.h.45268: tcp 1440
15:39:02.373678 IP e.f.g.h.45268 > a.b.c.d.80: tcp 0
15:39:02.373724 IP a.b.c.d.80 > e.f.g.h.45268: tcp 1440
15:39:02.373764 IP a.b.c.d.80 > e.f.g.h.45268: tcp 1440
15:39:02.380583 IP e.f.g.h.45268 > a.b.c.d.80: tcp 0
And this is the tcpdump as seen by the client on 192.168.0.22:
16:39:02.741810 IP 192.168.0.22.45268 > a.b.c.d.80: tcp 0
16:39:02.778115 IP a.b.c.d.80 > 192.168.0.22.45268: tcp 0
16:39:02.778152 IP 192.168.0.22.45268 > a.b.c.d.80: tcp 0
16:39:02.796612 IP 192.168.0.22.45268 > a.b.c.d.80: tcp 176
16:39:02.828845 IP a.b.c.d.80 > 192.168.0.22.45268: tcp 0
16:39:02.838956 IP a.b.c.d.80 > 192.168.0.22.45268: tcp 704
16:39:02.842058 IP a.b.c.d > 192.168.0.22: tcp
16:39:02.842083 IP 192.168.0.22.45268 > a.b.c.d.80: tcp 0
16:39:02.845211 IP a.b.c.d.80 > 192.168.0.22.45268: tcp 704
16:39:02.848334 IP a.b.c.d > 192.168.0.22: tcp
16:39:02.848358 IP 192.168.0.22.45268 > a.b.c.d.80: tcp 0
16:39:02.852771 IP a.b.c.d.80 > 192.168.0.22.45268: tcp 704
Unfortunately, the system clocks were not running synchronously.
As one can see, the three-way-handshake completes successfully, and
the client sends the http request. The server sends an ACK for the
request and begins with data transmission in packets with 1440 bytes
payload per packet. The client receives the big packet fragmented,
(that's the only way I can explain the undescribed datagrams on second
.842058 and .848334), and seems to either ACK or an error (does the
latter exist in TCP?). The server never sees the packet sent out by
the client, it also does not see any ICMP packets which might also
contain error messages. In the part of the tcpdump that is not shown
here, one sees that the server starts retransmits after sending like
32 KB (TCP transmission window?) and the connection stalls because no
ACKs arrive.
As soon as I turn off CBAC by removing "ip access-group" and "ip
inspect" statements from Vlan0 and Dialer0, all big downloads work fine.
As soon as CBAC is enabled again, the problem shows up again - even if
"permit ip any any" access lists and a simple inspect-Definition like
"ip inspect name TEST101 http" are chosen.
A few days later, I discovered "debug ip inspect tcp", and found debug
messages like
|000358: *Jan 23 18:08:08.375: CBAC* sis 83F69B20 L4 inspect result: SKIP packet 83FED8FC (server-ip:110) (LAN-IP:4155) bytes 138 ErrStr = Out-Of-Order Segment tcp
|000394: *Jan 23 18:08:08.559: CBAC* sis 83F69B20 L4 inspect result: DROP packet8396DAA4 (Dialer0-IP:4155) (server-ip:110) bytes 0 ErrStr = Invalid Ack (or no Ack) tcp
After reducing the tcp MSS on Vlan1 to 1300 ("ip tcp adjust-mss
1300"), downloads seem to work fine.
Can somebody explain why the problems show up when the mss is set to
the value recommended by the carrier?
Is this
(a) a configuration error?
(b) incompatibility of NAT with CBAC?
(c) incompatibility of the ISP, PMTUD and CBAC or
(d) an IOS bug?
Any hints will be appreciated.
Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835
More information about the cisco-nsp
mailing list