<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Are you able to provide a list of the IP addresses that being used for this reflection attack? I know that CloudFlare listed just the AS’es that has IPs in it and that helped some people get additional traction for resources to be applied to this issue. It’s one thing for their open NTP or DNS server to be on Jared Maruch’s lists, it’s another when they are actually used in an attack.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Frank<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Bryan Socha [mailto:bryan@serverstack.com] <br><b>Sent:</b> Saturday, March 08, 2014 11:25 AM<br><b>To:</b> Frank Bulk<br><b>Cc:</b> outages-discussion; Terrence<br><b>Subject:</b> Re: [Outages-discussion] [outages] enough of this ntp bs.<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal style='margin-bottom:12.0pt'>they are but it's taking out their backbones in the area, they are helping but they are literally blocking as far away as possible.. personally, the problem is the eyeball devices and their setup and how they ignore that they are the problem. This is getting really old really fast and it's time they do something about it. We all took care of business but they are the source of the problem and it's annoying.. <br><br>I know I'm just venting and not really on topic for outages but this is just nuts.. Things need to change and change fast. this might be starting with us but give it 3 weeks and you'll all be seeing it too....<o:p></o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'>btw if you don't know, this is about digitalocean, not serverstack, I just use this email on puck. btw, we're hiring if anyone is bored of their no attack comfy job that they get to go home at night and not work on the weekends.. <o:p></o:p></p></div><div><p class=MsoNormal><br clear=all><o:p></o:p></p><div><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#888888;background:white'>Bryan Socha</span></b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";background:white'><o:p></o:p></span></p><div><div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#666666;background:white'>Network Engineer<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#1155CC;background:white'>646.450.0472</span><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#666666;background:white'> | <u><a href="mailto:bryan@serverstack.com" target="_blank"><span style='color:#1155CC'>bryan@serverstack.com</span></a></u></span><span style='font-size:7.5pt;font-family:"Arial","sans-serif";background:white'><o:p></o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#888888;background:white'><o:p> </o:p></span></p></div><div><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#333333;background:white'>Server</span></b><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#FF6600;background:white'>Stack</span></b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#222222;background:white'> </span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#999999;background:white'>| Scale Big</span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#888888;background:white'><o:p></o:p></span></p></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p><div><p class=MsoNormal>On Sat, Mar 8, 2014 at 12:16 PM, Frank Bulk <<a href="mailto:frnkblk@iname.com" target="_blank">frnkblk@iname.com</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Forgive my naivety, but if the target port is the same (UDP 123) and there are only nine target IPs, why aren’t the upstream providers applying a simple filter upstream of “ip deny all <router ip> udp 123” ?</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Frank</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Bryan Socha [mailto:<a href="mailto:bryan@serverstack.com" target="_blank">bryan@serverstack.com</a>] <br><b>Sent:</b> Saturday, March 08, 2014 10:32 AM<br><b>To:</b> Terrence<br><b>Cc:</b> Frank Bulk; <a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a><br><b>Subject:</b> Re: [Outages-discussion] [outages] enough of this ntp bs.</span><o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>that won't help, their not attacking "me", they are attacking the ip address of all 9 provider links on my peering routers. I can't offload the cleaning, it's the datacenter itself under attack but on ips I can't even blackhole. I am at the mercy of providers to block their ip from being attacked without dropping my datacenter. 2 days ago we changed ips of the router and it took 45 seconds for the attack to move.. even if I had 100gbps links, the attack is still too large to stop.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#888888;background:white'>Bryan Socha</span></b><o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#666666;background:white'>Network Engineer</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#1155CC;background:white'><a href="tel:646.450.0472" target="_blank">646.450.0472</a></span><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#666666;background:white'> | <u><a href="mailto:bryan@serverstack.com" target="_blank"><span style='color:#1155CC'>bryan@serverstack.com</span></a></u></span><o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#888888;background:white'> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#333333;background:white'>Server</span></b><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#FF6600;background:white'>Stack</span></b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#222222;background:white'> </span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#999999;background:white'>| Scale Big</span><o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Sat, Mar 8, 2014 at 11:27 AM, Terrence <<a href="mailto:terrence.oconnor@gmail.com" target="_blank">terrence.oconnor@gmail.com</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Sounds like you need some DDoS help. Let me know. ;)<br><br>We've been certainly seeing an uptick in the number and size of attacks lately. I am not sure why the last mile providers aren't blocking spoofed source addresses.<br><br>There really isn't a good mitigation strategy other than offloading the attacks to a scalable provider. Or having ISPs validate the source prior to forwarding the packet. You just can't mitigate 450Gbps attacks at origin infrastructure.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br>-<o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Terrence<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Sent from my iPhone please excuse any errors.<o:p></o:p></p></div></div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><br>On Mar 8, 2014, at 11:10 AM, "Frank Bulk" <<a href="mailto:frnkblk@iname.com" target="_blank">frnkblk@iname.com</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>If you’ve seen more than 300 Gbps you should blog about it. =) The largest documented to date is CloudFlare’s. </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Are your upstream providers blocking NTP packets larger than a certain size?</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Frank</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Bryan Socha [<a href="mailto:bryan@serverstack.com" target="_blank">mailto:bryan@serverstack.com</a>] <br><b>Sent:</b> Saturday, March 08, 2014 10:04 AM<br><b>To:</b> Frank Bulk<br><b>Cc:</b> <a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a><br><b>Subject:</b> Re: [outages] enough of this ntp bs.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>It might sound like a joke but I've seen hundreds of gigs of attacks every morning. It'w all coming from home CPE devices and I think they need to start paying us for their incompetence. in 2014, why is this a problem!!?!???!?!?!!? it's time to be responsible. <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#888888;background:white'>Bryan Socha</span></b><o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#666666;background:white'>Network Engineer</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#1155CC;background:white'><a href="tel:646.450.0472" target="_blank">646.450.0472</a></span><span style='font-size:7.5pt;font-family:"Arial","sans-serif";color:#666666;background:white'> | <u><a href="mailto:bryan@serverstack.com" target="_blank"><span style='color:#1155CC'>bryan@serverstack.com</span></a></u></span><o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#888888;background:white'> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#333333;background:white'>Server</span></b><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#FF6600;background:white'>Stack</span></b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#222222;background:white'> </span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#999999;background:white'>| Scale Big</span><o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Sat, Mar 8, 2014 at 10:57 AM, Frank Bulk <<a href="mailto:frnkblk@iname.com" target="_blank">frnkblk@iname.com</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We’ve seen more DDoS attacks than normal, too, not just on ourselves, but on other networks where I have visibity. Funny that I saw an email in my inbox from Arbor Networks regarding an NTP DDoS webinar….</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Frank</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>] <b>On Behalf Of </b>Bryan Socha<br><b>Sent:</b> Saturday, March 08, 2014 9:41 AM<br><b>To:</b> <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br><b>Subject:</b> [outages] enough of this ntp bs.</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>all week long I'm seeing ntp attacks on provider ips on my router. Enough of this bs, it's time to stand up and block this BS....<o:p></o:p></p></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div></div></blockquote></div></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>_______________________________________________<br>Outages-discussion mailing list<br><a href="mailto:Outages-discussion@outages.org" target="_blank">Outages-discussion@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages-discussion" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><o:p></o:p></p></div></blockquote></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div></div></div></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>