<p dir="ltr">When did Windstream come back up after their big outage from the fiber cut? AS7029 has a couple thousand routes. Just a possible correlation I observed.</p>
<div class="gmail_quote">On Aug 12, 2014 4:12 PM, "Sean Hayes" <<a href="mailto:sean@engage.pro">sean@engage.pro</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">CIDR report is showing over 512k around the same time.</p>
<div class="gmail_quote">On Aug 12, 2014 4:09 PM, "Michael Hare via Outages" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
AS 2381 definitely saw Level3 sending > 512K for a brief time this AM, say 02:50 to 03:25. Perhaps those affected haven't figured it out yet. My below graph has timestamps in CDT.<br>
<br>
<a href="https://www.eng.wiscnet.net/stats/cgi-bin/genstatspage.fcgi?width=750&height=200&rrdfunc=MAX&rrdfile=%2Fvar%2Fflows%2Frrds%2FjnxBgpM2Prefixes%2Fdevice%3Dr-uwmadison-isp.wiscnet.net_family%3Dipv4_class%3Dunicast_ip%3D4.53.34.5_jnxBgpM2PrefixInPrefixesAccepted.rrd&time=CUSTOM&start_time=08%2F12%2F2014+2am&end_time=08%2F12%2F2014+6am" target="_blank">https://www.eng.wiscnet.net/<u></u>stats/cgi-bin/genstatspage.<u></u>fcgi?width=750&height=200&<u></u>rrdfunc=MAX&rrdfile=%2Fvar%<u></u>2Fflows%2Frrds%<u></u>2FjnxBgpM2Prefixes%2Fdevice%<u></u>3Dr-uwmadison-isp.wiscnet.net_<u></u>family%3Dipv4_class%3Dunicast_<u></u>ip%3D4.53.34.5_<u></u>jnxBgpM2PrefixInPrefixesAccept<u></u>ed.rrd&time=CUSTOM&start_time=<u></u>08%2F12%2F2014+2am&end_time=<u></u>08%2F12%2F2014+6am</a><br>
<br>
fwiw,<br>
-Michael<br>
<br>
On 8/12/2014 2:27 PM, Bill Wichers via Outages wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now.<br>
<br>
This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation.<br>
<br>
-Bill<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-----Original Message-----<br>
From: Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@<u></u>outages.org</a>] On Behalf Of<br>
Spencer Gaw via Outages<br>
Sent: Tuesday, August 12, 2014 3:20 PM<br>
To: Michael Peterman; <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
Subject: Re: [outages] So what is broken<br>
<br>
I would imagine it's related to the global routing table hitting 512k routes.<br>
<br>
<a href="http://www.cidr-report.org/cgi-" target="_blank">http://www.cidr-report.org/<u></u>cgi-</a><br>
bin/plot?file=%2fvar%2fdata%<u></u>2fbgp%2fas2.0%2fbgp%2dactive%<u></u>2etxt&start<br>
=1407179655&end=1407870855&<u></u>width=0%2e9&height=0%2e3&with=<u></u>step&y<br>
label=BGP+entries<br>
<br>
Regards,<br>
<br>
SG<br>
<br>
On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
So is this issue all related to a fiber cut or a DC/Peering point having issues?<br>
<br>
<a href="http://www.thewhir.com/web-hosting-news/liquidweb-among-" target="_blank">http://www.thewhir.com/web-<u></u>hosting-news/liquidweb-among-</a><br>
</blockquote>
companies-affe<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
cted-major-outage-across-us-<u></u>network-providers<br>
<br>
<br>
<br>
Michael Peterman<br>
<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@<u></u>outages.org</a>] On Behalf Of<br>
<a href="mailto:outages-request@outages.org" target="_blank">outages-request@outages.org</a><br>
Sent: Tuesday, August 12, 2014 11:00 AM<br>
To: <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
Subject: Outages Digest, Vol 75, Issue 9<br>
<br>
Send Outages mailing list submissions to<br>
<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/<u></u>mailman/listinfo/outages</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:outages-request@outages.org" target="_blank">outages-request@outages.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:outages-owner@outages.org" target="_blank">outages-owner@outages.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific than "Re:<br>
</blockquote>
Contents of Outages digest..."<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Today's Topics:<br>
<br>
1. Windstream (Mike Walter)<br>
2. CoreSite Any2 LA (jd)<br>
3. Re: CoreSite Any2 LA (Paul Stewart)<br>
4. LastPass.com has a DC outage (Chris Egeland)<br>
5. Major outages today, not much info at this time<br>
(Hoyle Anderson (AM))<br>
6. Re: Major outages today, not much info at this time (Teun Vink)<br>
<br>
<br>
------------------------------<u></u>------------------------------<u></u>----------<br>
<br>
Message: 1<br>
Date: Tue, 12 Aug 2014 13:19:08 +0000<br>
From: Mike Walter <<a href="mailto:mwalter@3z.net" target="_blank">mwalter@3z.net</a>><br>
To: "'<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>'" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br>
Subject: [outages] Windstream<br>
Message-ID:<br>
<br>
</blockquote>
<<u></u>D7092CF606B8964193F248C92106B7<u></u>810FCDE655@zionex01.ZION.loc<br>
al><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky<br>
</blockquote>
customers, no ETA to repair based on their phone message. Still on hold<br>
waiting to open case to find out more.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-Mike<br>
<br>
-------------- next part -------------- An HTML attachment was<br>
scrubbed...<br>
URL:<br>
<br>
</blockquote>
<<a href="https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c" target="_blank">https://puck.nether.net/<u></u>pipermail/outages/attachments/<u></u>20140812/194c6c</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
4b/attachment-0001.html><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Tue, 12 Aug 2014 07:19:04 -0700<br>
From: jd <<a href="mailto:jd@boldnetworks.net" target="_blank">jd@boldnetworks.net</a>><br>
To: "'<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>'" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br>
Subject: [outages] CoreSite Any2 LA<br>
Message-ID: <<a href="mailto:53EA2258.2020601@boldnetworks.net" target="_blank">53EA2258.2020601@<u></u>boldnetworks.net</a>><br>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br>
<br>
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing<br>
</blockquote>
some things, if anyone could debunk.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks<br>
JD<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Tue, 12 Aug 2014 10:42:39 -0400<br>
From: Paul Stewart <<a href="mailto:paul@paulstewart.org" target="_blank">paul@paulstewart.org</a>><br>
To: "'<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>'" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br>
Subject: Re: [outages] CoreSite Any2 LA<br>
Message-ID: <<a href="mailto:D00FA00A.806C1%25paul@paulstewart.org" target="_blank">D00FA00A.806C1%paul@<u></u>paulstewart.org</a>><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
No issues here at this point?. At least that have become evident?<br>
<br>
Paul<br>
<br>
<br>
<br>
On 2014-08-12, 10:19 AM, "jd via Outages" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is anyone having issues with Any2 in LA? I have no evidence but I'm<br>
hearing some things, if anyone could debunk.<br>
<br>
Thanks<br>
JD<br>
______________________________<u></u>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/<u></u>mailman/listinfo/outages</a><br>
<br>
</blockquote>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Tue, 12 Aug 2014 15:04:58 +0000<br>
From: Chris Egeland <<a href="mailto:chris.egeland@ferrotherm.com" target="_blank">chris.egeland@ferrotherm.com</a>><br>
To: "<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a> List" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br>
Subject: [outages] LastPass.com has a DC outage<br>
Message-ID: <<u></u>EA49DAFC5FDC1A40B16DEE2D6ED546<u></u>9A9FD277AB@mail01><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
<a href="https://twitter.com/LastPass/status/499199566701658112" target="_blank">https://twitter.com/LastPass/<u></u>status/499199566701658112</a><br>
<br>
"Update: One of our data centers went down, we have the service fully<br>
</blockquote>
running on another data center & are working to minimize impact."<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'm getting "An error occurred while attempting to contact the server.<br>
</blockquote>
Please check your Internet connection." In my LastPass bar on Firefox.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So, if you're getting errors with LastPass, this is why.<br>
<br>
<br>
<br>
______________________________<u></u>__<br>
<br>
This e-mail and any attachments may contain privileged or confidential<br>
</blockquote>
information of Ferrotherm Corporation. This e-mail is intended solely for the<br>
use of the individual or entity to which it is addressed. If you are not the<br>
intended recipient of this e-mail, you are hereby notified that any copying,<br>
distribution, dissemination or action taken in relation to the contents of this<br>
e-mail and any of its attachments is strictly prohibited and may be unlawful. If<br>
you have received this e-mail in error, please notify the sender immediately<br>
and permanently delete the original e-mail and destroy any copies or<br>
printouts of this e-mail as well as any attachments.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-------------- next part -------------- An HTML attachment was<br>
scrubbed...<br>
URL:<br>
<br>
</blockquote>
<<a href="https://puck.nether.net/pipermail/outages/attachments/20140812/837627" target="_blank">https://puck.nether.net/<u></u>pipermail/outages/attachments/<u></u>20140812/837627</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
0e/attachment-0001.html><br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Tue, 12 Aug 2014 15:20:45 +0000<br>
From: "Hoyle Anderson (AM)" <<a href="mailto:hoyle.anderson@dimensiondata.com" target="_blank">hoyle.anderson@dimensiondata.<u></u>com</a>><br>
To: "<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br>
Subject: [outages] Major outages today, not much info at this time<br>
Message-ID:<br>
<br>
<br>
</blockquote>
<<u></u>7C6FA1D8224C7944A005E6587BDEA5<u></u><a href="mailto:0A03F692@usispsvexdb03.na" target="_blank">0A03F692@usispsvexdb03.na</a>.<u></u>didata.lo<br>
cal><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Content-Type: text/plain; charset="utf-8"<br>
<br>
I know this isn?t much help, but there are major problems with multiple<br>
</blockquote>
ISPs since around 4-5 AM EST. I really don?t have much detail, but I have<br>
sites that are unreachable from some providers. Looks like Comcast, level3,<br>
ATT, cogent, etc.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard<br>
</blockquote>
one report of a datacenter outage.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Sources:<br>
<a href="http://downdetector.com/" target="_blank">http://downdetector.com/</a><br>
<a href="https://twitter.com/liquidweb" target="_blank">https://twitter.com/liquidweb</a><br>
<br>
-------------- next part -------------- An HTML attachment was<br>
scrubbed...<br>
URL:<br>
<br>
</blockquote>
<<a href="https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8" target="_blank">https://puck.nether.net/<u></u>pipermail/outages/attachments/<u></u>20140812/fa57b8</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
50/attachment-0001.html><br>
<br>
------------------------------<br>
<br>
Message: 6<br>
Date: Tue, 12 Aug 2014 17:42:05 +0200<br>
From: Teun Vink <<a href="mailto:teun@teun.tv" target="_blank">teun@teun.tv</a>><br>
To: "Hoyle Anderson (AM)" <<a href="mailto:hoyle.anderson@dimensiondata.com" target="_blank">hoyle.anderson@dimensiondata.<u></u>com</a>><br>
Cc: "<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><br>
Subject: Re: [outages] Major outages today, not much info at this time<br>
Message-ID: <1407858125.27157.21.camel@<u></u>moridin><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I know this isn?t much help, but there are major problems with<br>
multiple ISPs since around 4-5 AM EST. I really don?t have much<br>
detail, but I have sites that are unreachable from some providers.<br>
Looks like Comcast, level3, ATT, cogent, etc.<br>
<br>
<br>
<br>
So, it?s probably not just you, but I?m afraid I don?t know who it is.<br>
I heard one report of a datacenter outage.<br>
<br>
</blockquote>
Hi,<br>
<br>
Some routing tables hit 512K routes today. Some old hardware and<br>
</blockquote>
software can't handle that and either crash or ignore newly learned routes.<br>
So this may cause some disturbances in the force.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
HTH,<br>
Teun<br>
<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
______________________________<u></u>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/<u></u>mailman/listinfo/outages</a><br>
<br>
<br>
------------------------------<br>
<br>
End of Outages Digest, Vol 75, Issue 9<br>
******************************<u></u>********<br>
<br>
______________________________<u></u>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/<u></u>mailman/listinfo/outages</a><br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/<u></u>mailman/listinfo/outages</a><br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/<u></u>mailman/listinfo/outages</a><br>
<br>
</blockquote>
______________________________<u></u>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/<u></u>mailman/listinfo/outages</a><br>
</blockquote></div>
</blockquote></div>