[c-nsp] DDOS, router acted "oddly".

[mack]

> Message: 3
> Date: Thu, 30 Aug 2007 10:32:05 -0400
> From: Drew Weaver <drew.weaver at thenap.com>
> Subject: Re: [c-nsp] DDOS, router acted "oddly".
> To: "cisco-nsp at puck.nether.net" <cisco-nsp at puck.nether.net>
> Message-ID:
>
> <B7152C470C9BF3448ED33F16A75D81C14D0B8DA2DD at exchanga.thenap.com>
> Content-Type: text/plain; charset="us-ascii"
>
> More information, the traffic they sent looked like this:
>
> 1188461504.873821 y.y.y.y -> x.x.x.x UDP Source port: 45362
> Destination port: 11067[Malformed Packet]
>
> 0000  00 18 8b 4e bf df 00 05 dd 27 58 40 08 00 45 00
> ...N.....'X at ..E.
> 0010  00 1d 00 00 40 00 38 11 94 c9 c1 1b 56 c5 d1 33
> .... at .8.....V..3
> 0020  c4 f2 b1 32 2b 3b 00 09 45 67 30 00 00 00 00 00
> ...2+;..Eg0.....
> 0030  00 00 00 00 00 00 00 00 00 00 00 00               ............
>

If netflow is activated, UDP floods will have a negative effect on the 6509 platform
regardless of supervisor unit.  As the packets show as malformed they may also have
been getting shunted to a software path on the 6509.  Hardware rate limiters and
control plane policing are your friend in these situations.

The GSRs may have had issues as well but my prime suspect would be the 6509.
The 6509 may have been throttling traffic causing the GSR buffers to back up.
Look at the 'show proc cpu hist'.
See if the average spiked significantly during the attack.

> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-
> bounces at puck.nether.net] On Behalf Of Drew Weaver
> Sent: Thursday, August 30, 2007 9:52 AM
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] DDOS, router acted "oddly".
>
>         I believe I know why I had the issue I had last evening when a
> 500Mbps DDOS hit our network. I believe it is due to queuing issues,
> but I am not sure, I wanted to ask you folks what you thought.
>
> The topology of the 'attack ' is as such:
>
> Attacker - Internet - 3Gbps aggregate(4 connections) - 2x Cisco GSR
> 12000 - 4x Gig-E - Catalyst 6509 - 100Mbps -- target host
>
> So last evening we were hit with a 500Mbps DDOS attack, this shouldn't
> have been a big deal as we have over 3Gbps in aggregate bandwidth and
> this 500Mbps pushed our total utilization up to around 1300Mbps.
> However, we noticed that the DDOS was degrading connectivity for all
> hosts on the network.
>
> * The (multiple) gig-e connections between the GSRs and the Catalyst
> 6509 were nowhere near their maximum capacity
> * I see no errors in the log files of either of the two GSRs which were
> involved
> * The 100Mbps port which the target host was connected to was obviously
> pegged.
> * There were no errors logged on that particular catalyst (although I
> believe the problem is obviously with the GSRs)
>
> I don't really see any "good?" reason why all of the traffic flowing
> through both of the GSR 12ks would have been reduced to a crawl unless
> there was some kind of queue backlash between the Catalyst and the GSR
> 12ks.
>
> Does anyone have any advice or insight?
>
> Thanks,
> -Drew
>

LR Mack McBride
Network Administrator
Alpha Red, Inc.