[c-nsp] 6509-IOS Firewalls causing RST storm that max CPU

Grant Moerschel gm at wavegard.com
Sat Feb 12 23:47:24 EST 2005 

You are right. I didn't notice that that capture showed it as port 80. 
So my statement of it being ftp is erroneous.  Here is another capture 
but it does not show the beginning of the flow.  I don't happen to have 
another capture right now.


16:08:06.657390 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.657480 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.657644 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.657744 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320
16:08:06.657856 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.657947 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.658111 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.658212 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320
16:08:06.658322 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.658411 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.658576 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.658677 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320
16:08:06.658783 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.658872 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.659040 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.659141 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320
16:08:06.659246 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.659338 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.659506 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.659608 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320
16:08:06.659714 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.659804 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.659986 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.660086 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320
16:08:06.660191 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.660282 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.660895 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.661003 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320
16:08:06.661104 10.232.186.100.25 > 10.237.102.68.56095: R 0:0(0) win 5840
16:08:06.661194 10.232.166.15.21 > 10.237.102.81.33488: R 0:0(0) win 5840
16:08:06.661394 10.237.102.68.56095 > 10.232.186.100.25: R 0:0(0) win 5792
16:08:06.661496 10.237.102.81.33488 > 10.232.166.15.21: R 0:0(0) win 50320

Richard Golodner wrote:
> Hi Grant, look at the ports that were in use. Why would you be sending over
> 80? Could you post some sanitized packet captures for me? The thing that
> hangs me is the absence of traffic completely on your trunked vlan. Hope we
> can help.
> 						Richard Golodner
> 
> -----Original Message-----
> From: Grant Moerschel [mailto:gm at wavegard.com]
> Sent: Saturday, February 12, 2005 7:27 PM
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] 6509-IOS Firewalls causing RST storm that max CPU
> 
> 
> Hello all,
> I have an environment with two 6509 switches linked together via a trunk 
> passing one vlan.  Both chassis run 12.2.18sxd3 IOS firewall code.  We 
> use ip inspect inbound on both sides of the vlan providing the link.  We 
> are seeing strange behavior (often with ftp) that when the client ends a 
> connection correctly using fins (fin, then fin, then ack, then ack), the 
> server will shortly thereafter send a rst to the client which will kick 
> off a rst storm ping pong match back and forth between the client and 
> server to the point that it makes one sup2 go to 99% cpu and the sup720 
> on the other side go to 80%.
> 
> Here's the strange thing.  The hosts are not doing it but the firewall 
> function is. This was proven with sniffer traces showing zero traffic 
> actually hitting the vlans where the hosts were during a storm. We think 
> that the firewall thinks the host is under attack and is resetting the 
> connection on the hosts behalf.  Bear in mind that this ios rev does 
> _not_ support IP audit which is part of the cbac functionality. IP audit 
> will allow you, on smaller platforms, to send rst packets in response to 
> attacks. So this 6509 shouldn't even be doing this unless I am 
> mistaken??  What am I missing?
> 
> I bumped the fin wait time to 15 seconds. Fin wait keeps the session in 
> the firewall for x seconds after a session is torn down so that any 
> other packets coming in won't ge bounced. Used with NFS I think which is 
> not what we are doing but it was worth a try.
> 
> Here's a snippet from a flood I sniffed - starting at the end of a 
> normal tcp session. Notice where the rst start going back and forth:
> 
> ---
> 16:57:32.246355 IP 10.237.102.53.50270 > 10.232.188.200.80: S 
> 4081749102:4081749102(0) win 5840 <mss 1460,sackOK,timestamp 105526533 
> 0,nop,wscale 0>
> 16:57:32.286018 IP 10.232.188.200.80 > 10.237.102.53.50270: S 
> 482684002:482684002(0) ack 4081749103 win 16464 <mss 1372,nop,wscale 
> 0,nop,nop,timestamp 0 0,nop,nop,sackOK>
> 16:57:32.286185 IP 10.237.102.53.50270 > 10.232.188.200.80: . ack 1 win 
> 5840 <nop,nop,timestamp 105526537 0>
> 16:57:32.288077 IP 10.237.102.53.50270 > 10.232.188.200.80: P 1:329(328) 
> ack 1 win 5840 <nop,nop,timestamp 105526537 0>
> 16:57:32.288157 IP 10.237.102.53.50270 > 10.232.188.200.80: P 
> 329:635(306) ack 1 win 5840 <nop,nop,timestamp 105526537 0>
> 16:57:32.327566 IP 10.232.188.200.80 > 10.237.102.53.50270: . ack 635 
> win 15830 <nop,nop,timestamp 12111000 105526537>
> 16:57:32.476047 IP 10.232.188.200.80 > 10.237.102.53.50270: P 1:172(171) 
> ack 635 win 15830 <nop,nop,timestamp 12111002 105526537>
> 16:57:32.476249 IP 10.237.102.53.50270 > 10.232.188.200.80: . ack 172 
> win 6432 <nop,nop,timestamp 105526556 12111002>
> 16:57:32.478250 IP 10.232.188.200.80 > 10.237.102.53.50270: . 
> 172:1532(1360) ack 635 win 15830 <nop,nop,timestamp 12111002 105526537>
> 16:57:32.478316 IP 10.237.102.53.50270 > 10.232.188.200.80: . ack 1532 
> win 9520 <nop,nop,timestamp 105526556 12111002>
> 16:57:32.518355 IP 10.232.188.200.80 > 10.237.102.53.50270: P 
> 2892:3203(311) ack 635 win 15830 <nop,nop,timestamp 12111002 105526556>
> 16:57:32.518503 IP 10.237.102.53.50270 > 10.232.188.200.80: . ack 1532 
> win 9520 <nop,nop,timestamp 105526560 12111002,nop,nop,sack sack 1 
> {2892:3203} >
> 16:57:32.518889 IP 10.232.188.200.80 > 10.237.102.53.50270: . 
> 1532:2892(1360) ack 635 win 15830 <nop,nop,timestamp 12111002 105526556>
> 16:57:32.518939 IP 10.237.102.53.50270 > 10.232.188.200.80: . ack 3203 
> win 12240 <nop,nop,timestamp 105526560 12111002>
> 16:57:32.519095 IP 10.237.102.53.50270 > 10.232.188.200.80: F 635:635(0) 
> ack 3203 win 12240 <nop,nop,timestamp 105526560 12111002>
> 16:57:32.560448 IP 10.232.188.200.80 > 10.237.102.53.50270: F 
> 3203:3203(0) ack 635 win 15830 <nop,nop,timestamp 12111003 105526560>
> 16:57:32.560464 IP 10.232.188.200.80 > 10.237.102.53.50270: . ack 636 
> win 15830 <nop,nop,timestamp 12111003 105526560>
> 16:57:32.560629 IP 10.237.102.53.50270 > 10.232.188.200.80: . ack 3204 
> win 12240 <nop,nop,timestamp 105526564 12111003>
> 16:57:32.600087 IP 10.232.188.200.80 > 10.237.102.53.50270: R 
> 482687205:482687205(0) win 0
> 16:57:32.600394 IP 10.237.102.53.50270 > 10.232.188.200.80: R 
> 482687205:482687205(0) win 15830
> 16:57:32.600581 IP 10.232.188.200.80 > 10.237.102.53.50270: R 0:0(0) win 
> 12240
> 16:57:32.600774 IP 10.237.102.53.50270 > 10.232.188.200.80: R 0:0(0) win 
> 15830
> 16:57:32.600953 IP 10.232.188.200.80 > 10.237.102.53.50270: R 0:0(0) win 
> 12240
> 16:57:32.601126 IP 10.237.102.53.50270 > 10.232.188.200.80: R 0:0(0) win 
> 15830
> 16:57:32.601298 IP 10.232.188.200.80 > 10.237.102.53.50270: R 0:0(0) win 
> 12240
> 16:57:32.601476 IP 10.237.102.53.50270 > 10.232.188.200.80: R 0:0(0) win 
> 15830
> 16:57:32.601649 IP 10.232.188.200.80 > 10.237.102.53.50270: R 0:0(0) win 
> 12240
> 16:57:32.601831 IP 10.237.102.53.50270 > 10.232.188.200.80: R 0:0(0) win 
> 15830
> 16:57:32.602002 IP 10.232.188.200.80 > 10.237.102.53.50270: R 0:0(0) win 
> 12240
> 16:57:32.602184 IP 10.237.102.53.50270 > 10.232.188.200.80: R 0:0(0) win 
> 15830
> ... and it keeps going ...
> 
> Grant Moerschel
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

-- 
==================================================
Grant P. Moerschel CISSP, CCSP, CCNP, CWNA
WaveGard, Inc.
Information Security Solutions
Consulting * Training * Integration

+1.703.568.5077 * gm at wavegard.com

Questions on which certification is which?...
Visit http://www.wavegard.com/certifications.html
==================================================

More information about the cisco-nsp mailing list