[outages] So what is broken

Tue Aug 12 15:35:32 EDT 2014

AS 2381 definitely saw Level3 sending > 512K for a brief time this AM, 
say 02:50 to 03:25.  Perhaps those affected haven't figured it out yet. 
  My below graph has timestamps in CDT.

https://www.eng.wiscnet.net/stats/cgi-bin/genstatspage.fcgi?width=750&height=200&rrdfunc=MAX&rrdfile=%2Fvar%2Fflows%2Frrds%2FjnxBgpM2Prefixes%2Fdevice%3Dr-uwmadison-isp.wiscnet.net_family%3Dipv4_class%3Dunicast_ip%3D4.53.34.5_jnxBgpM2PrefixInPrefixesAccepted.rrd&time=CUSTOM&start_time=08%2F12%2F2014+2am&end_time=08%2F12%2F2014+6am

fwiw,
-Michael

On 8/12/2014 2:27 PM, Bill Wichers via Outages wrote:
> I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now.
>
> This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation.
>
>    -Bill
>
>
>> -----Original Message-----
>> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
>> Spencer Gaw via Outages
>> Sent: Tuesday, August 12, 2014 3:20 PM
>> To: Michael Peterman; outages at outages.org
>> Subject: Re: [outages] So what is broken
>>
>> I would imagine it's related to the global routing table hitting 512k routes.
>>
>> http://www.cidr-report.org/cgi-
>> bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start
>> =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y
>> label=BGP+entries
>>
>> Regards,
>>
>> SG
>>
>> On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
>>> So is this issue all related to a fiber cut or a DC/Peering point having issues?
>>>
>>> http://www.thewhir.com/web-hosting-news/liquidweb-among-
>> companies-affe
>>> cted-major-outage-across-us-network-providers
>>>
>>>
>>>
>>> Michael  Peterman
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
>>> outages-request at outages.org
>>> Sent: Tuesday, August 12, 2014 11:00 AM
>>> To: outages at outages.org
>>> Subject: Outages Digest, Vol 75, Issue 9
>>>
>>> Send Outages mailing list submissions to
>>> 	outages at outages.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> 	https://puck.nether.net/mailman/listinfo/outages
>>> or, via email, send a message with subject or body 'help' to
>>> 	outages-request at outages.org
>>>
>>> You can reach the person managing the list at
>>> 	outages-owner at outages.org
>>>
>>> When replying, please edit your Subject line so it is more specific than "Re:
>> Contents of Outages digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>      1. Windstream (Mike Walter)
>>>      2. CoreSite Any2 LA (jd)
>>>      3. Re: CoreSite Any2 LA (Paul Stewart)
>>>      4. LastPass.com has a DC outage (Chris Egeland)
>>>      5. Major outages today, not much info at this time
>>>         (Hoyle Anderson (AM))
>>>      6. Re: Major outages today, not much info at this time (Teun Vink)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Tue, 12 Aug 2014 13:19:08 +0000
>>> From: Mike Walter <mwalter at 3z.net>
>>> To: "'outages at outages.org'" <outages at outages.org>
>>> Subject: [outages] Windstream
>>> Message-ID:
>>>
>> 	<D7092CF606B8964193F248C92106B7810FCDE655 at zionex01.ZION.loc
>> al>
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky
>> customers, no ETA to repair based on their phone message.  Still on hold
>> waiting to open case to find out more.
>>> -Mike
>>>
>>> -------------- next part -------------- An HTML attachment was
>>> scrubbed...
>>> URL:
>>>
>> <https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c
>>> 4b/attachment-0001.html>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Tue, 12 Aug 2014 07:19:04 -0700
>>> From: jd <jd at boldnetworks.net>
>>> To: "'outages at outages.org'" <outages at outages.org>
>>> Subject: [outages] CoreSite Any2 LA
>>> Message-ID: <53EA2258.2020601 at boldnetworks.net>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing
>> some things, if anyone could debunk.
>>>
>>> Thanks
>>> JD
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 3
>>> Date: Tue, 12 Aug 2014 10:42:39 -0400
>>> From: Paul Stewart <paul at paulstewart.org>
>>> To: "'outages at outages.org'" <outages at outages.org>
>>> Subject: Re: [outages] CoreSite Any2 LA
>>> Message-ID: <D00FA00A.806C1%paul at paulstewart.org>
>>> Content-Type: text/plain;	charset="UTF-8"
>>>
>>> No issues here at this point?. At least that have become evident?
>>>
>>> Paul
>>>
>>>
>>>
>>> On 2014-08-12, 10:19 AM, "jd via Outages" <outages at outages.org> wrote:
>>>
>>>> Is anyone having issues with Any2 in LA? I have no evidence but I'm
>>>> hearing some things, if anyone could debunk.
>>>>
>>>> Thanks
>>>> JD
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 4
>>> Date: Tue, 12 Aug 2014 15:04:58 +0000
>>> From: Chris Egeland <chris.egeland at ferrotherm.com>
>>> To: "outages at outages.org List" <outages at outages.org>
>>> Subject: [outages] LastPass.com has a DC outage
>>> Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB at mail01>
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> https://twitter.com/LastPass/status/499199566701658112
>>>
>>> "Update: One of our data centers went down, we have the service fully
>> running on another data center & are working to minimize impact."
>>>
>>> I'm getting "An error occurred while attempting to contact the server.
>> Please check your Internet connection." In my LastPass bar on Firefox.
>>>
>>> So, if you're getting errors with LastPass, this is why.
>>>
>>>
>>>
>>> ________________________________
>>>
>>> This e-mail and any attachments may contain privileged or confidential
>> information of Ferrotherm Corporation. This e-mail is intended solely for the
>> use of the individual or entity to which it is addressed. If you are not the
>> intended recipient of this e-mail, you are hereby notified that any copying,
>> distribution, dissemination or action taken in relation to the contents of this
>> e-mail and any of its attachments is strictly prohibited and may be unlawful. If
>> you have received this e-mail in error, please notify the sender immediately
>> and permanently delete the original e-mail and destroy any copies or
>> printouts of this e-mail as well as any attachments.
>>> -------------- next part -------------- An HTML attachment was
>>> scrubbed...
>>> URL:
>>>
>> <https://puck.nether.net/pipermail/outages/attachments/20140812/837627
>>> 0e/attachment-0001.html>
>>>
>>> ------------------------------
>>>
>>> Message: 5
>>> Date: Tue, 12 Aug 2014 15:20:45 +0000
>>> From: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>>> To: "outages at outages.org" <outages at outages.org>
>>> Subject: [outages] Major outages today, not much info at this time
>>> Message-ID:
>>>
>>>
>> <7C6FA1D8224C7944A005E6587BDEA50A03F692 at usispsvexdb03.na.didata.lo
>> cal>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> I know this isn?t much help, but there are major problems with multiple
>> ISPs since around 4-5 AM EST.  I really don?t have much detail, but I have
>> sites that are unreachable from some providers.  Looks like Comcast, level3,
>> ATT, cogent, etc.
>>>
>>> So, it?s probably not just you, but I?m afraid I don?t know who it is.  I heard
>> one report of a datacenter outage.
>>>
>>> Sources:
>>> http://downdetector.com/
>>> https://twitter.com/liquidweb
>>>
>>> -------------- next part -------------- An HTML attachment was
>>> scrubbed...
>>> URL:
>>>
>> <https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8
>>> 50/attachment-0001.html>
>>>
>>> ------------------------------
>>>
>>> Message: 6
>>> Date: Tue, 12 Aug 2014 17:42:05 +0200
>>> From: Teun Vink <teun at teun.tv>
>>> To: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>>> Cc: "outages at outages.org" <outages at outages.org>
>>> Subject: Re: [outages] Major outages today, not much info at this time
>>> Message-ID: <1407858125.27157.21.camel at moridin>
>>> Content-Type: text/plain; charset="UTF-8"
>>>
>>> On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
>>>> I know this isn?t much help, but there are major problems with
>>>> multiple ISPs since around 4-5 AM EST.  I really don?t have much
>>>> detail, but I have sites that are unreachable from some providers.
>>>> Looks like Comcast, level3, ATT, cogent, etc.
>>>>
>>>>
>>>>
>>>> So, it?s probably not just you, but I?m afraid I don?t know who it is.
>>>> I heard one report of a datacenter outage.
>>>>
>>> Hi,
>>>
>>> Some routing tables hit 512K routes today. Some old hardware and
>> software can't handle that and either crash or ignore newly learned routes.
>> So this may cause some disturbances in the force.
>>>
>>> HTH,
>>> Teun
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>> ------------------------------
>>>
>>> End of Outages Digest, Vol 75, Issue 9
>>> **************************************
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>