[nsp-sec] 50Gbps NTP Attack, 23377 Source IP addresses

Mon Mar 3 09:57:05 EST 2014

I was wrong! That is I reported that the NTP4 get_monlist1 which has a standard frame of 234 (220 ip packet) wasn't the main trigger packet we are seeing in attacks.

Then I found what I believe to be the tool or rather one that is really close.
If someone REALLY needs the tool let me know I can share that or the pcaps as needed internal only ...

> ntpdos.py has the get_monlist1 command in it.
> 
> 
> 
> ntp.py doesn't. I think mostly we are seeing this one.
I shared it with a coworker John Schiel (some of you met him or saw him at MA3WWG and he ran it here are the results.
It is NOT the 234 (or 220) get_monlist1 that others have reported. It is 36 bytes.

John Schiel wrote the following.

"I ran ntp.py.

I took the ntp.py script and found a vulnerable IP, 67.148.184.18, and ran the script. I first tried to find one of our IPs but couldn't find an open IP so quickly. I would like to have a closed vulnerable ntpd that I can use for future tests.

Click on the IP header, it's 36 bytes and it has the MON_GETLIST_1 request code. 

The response includes 100 replies at 468 bytes each, quite an amplification.  46800/36 = 1300X.

I tried to save the output of ntp.py but it comes back as binary so I can't really see what's in there yet. All I do see when I run the script is: 

"~/scripts$ python ntp.py
67.148.184.18(123) said "<insert binary garbage here>

If you want the file output when I ran the command, let me know and I'll send it.

I also ran the "ntpdc -c monlist" command against the same IP and got a huge response. I'm including that also.

So, good find Don. I'm sure with a little more coding we could force the source port to be 80, spoof the src IP, and replicate the issue exactly.

--John "

Upgrading is still not the most effective way to stop this. BCP38/84 is. Next to that limiting whom can talk to your NTP services (limit all of your reflective surfaces, DNS, NTP, chargen...).

My 13x amplification (36 : 468) is still valid for a single packet but as John points out above you may get as many as 100 of those providing a 1300x amplification.

get_monlist1 NTPv2 IS being used but not the 234 byte monlist NTPv4 that others have been suggesting. We see a LITTLE of that but the main culprit is the 36 byte one which came from a tool like the one John ran or something close.

NEW: After much research we are now recommending rate limiting udp src port 123 packetsize 468 (or the initial packet which is 36 bytes towards udp 123).
Normal NTP timesync packets will generally be 76 bytes so you could turn this around and rate limit ALL udp/123 source or dst which isn't 76 octets. I know of at least one other ISP that has had success with that.

I have NOT captured the actual attacks in a pcap so this is based on netflow and LAB NTP packet lab analysis.

(coffee != sleep) & (!coffee == sleep)
 Donald.Smith at centurylink.com

From: nsp-security [nsp-security-bounces at puck.nether.net] on behalf of Joel L. Rosenblatt [joel at columbia.edu]
Sent: Sunday, March 02, 2014 10:04 AM
To: Phil Rosenthal
Cc: NSP-SEC List
Subject: Re: [nsp-sec] 50Gbps NTP Attack, 23377 Source IP addresses

----------- nsp-security Confidential --------

Hi Phil,

I have been trying to get our machines cleaned also (a little like
herding cats :-)

There has be some discussion about monlist not being the only attack
mechanism - have you made any progress with determining if this is the
primary attack vector or not?

Sorry for the attacks, but thanks for putting the time into this

Regards,
Joel

Joel Rosenblatt, Director Network & Computer Security
Columbia Information Security Office (CISO)
Columbia University, 612 W 115th Street, NY, NY 10025 / 212 854 3033
http://www.columbia.edu/~joel
Public PGP key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90BD740BCC7326C3

On Sun, Mar 2, 2014 at 11:02 AM, Phil Rosenthal <pr at isprime.com> wrote:
> ----------- nsp-security Confidential --------
>
>
> On Mar 2, 2014, at 9:53 PM, Brian J Smith-Sweeney <bsmithsweeney at nyu.edu> wrote:
>
>> 1) Kicking anyone off the network that's reported to us as being abused.  We used to just reserve this for compromised systems but feel it's appropriate when we're sourcing DoS (particularly ntp, which has gone from a rare to nearly daily occurrence).
>> 2) Scanning the network for not-yet-abused systems (nmap with ntp-monlist script) .  This is fraught with the standard perils of scanning for udp but it's turned up a bunch of hosts we didn't already know about.  I've got a short Python script to parse nmap's xml output for vulnerable systems.
>> 3) Notify vulnerable systems with a sunset date - fix or off by X time.
>>
>> I'm sharing this because I know folks sometimes find it useful to have peers to point at to help justify action to management.  If that's you, feel free to tell yours this is what we're doing.
>>
>> I'm talking to our NOC team about what are capabilities are for handling this at the border, but we don't do much filtering now, and it's tricky.
>>
>
> I agree that your approach is the most sensible one.
>
> Filtering at the border is a bad precedent.  A world with unreliable NTP is going to cause 'mysterious' problems in all sorts of unexpected places.  Let's try to avoid throwing the baby out with the bath water and just fix the NTP DRDoS problem without making new ones.
>
> Also -- Thanks to you and everyone who has worked to clean up NTP reflectors on their network.  We received another attack today -- 90% smaller.  I am hopeful that this indicates that 90% of the list I posted has already been cleaned. [Though, you know how these things go, and tomorrow may be an even larger attack]
>
> Regards,
> -Phil
>
>
> _______________________________________________
> nsp-security mailing list
> nsp-security at puck.nether.net
> https://puck.nether.net/mailman/listinfo/nsp-security
>
> Please do not Forward, CC, or BCC this E-mail outside of the nsp-security
> community. Confidentiality is essential for effective Internet security counter-measures.
> _______________________________________________

_______________________________________________
nsp-security mailing list
nsp-security at puck.nether.net
https://puck.nether.net/mailman/listinfo/nsp-security

Please do not Forward, CC, or BCC this E-mail outside of the nsp-security
community. Confidentiality is essential for effective Internet security counter-measures.
_______________________________________________