[outages] So what is broken

Sean Hayes sean at engage.pro
Tue Aug 12 16:19:13 EDT 2014 

When did Windstream come back up after their big outage from the fiber
cut?  AS7029 has a couple thousand routes.  Just a possible correlation I
observed.
On Aug 12, 2014 4:12 PM, "Sean Hayes" <sean at engage.pro> wrote:

> CIDR report is showing over 512k around the same time.
> On Aug 12, 2014 4:09 PM, "Michael Hare via Outages" <outages at outages.org>
> wrote:
>
>> AS 2381 definitely saw Level3 sending > 512K for a brief time this AM,
>> say 02:50 to 03:25.  Perhaps those affected haven't figured it out yet.  My
>> below graph has timestamps in CDT.
>>
>> https://www.eng.wiscnet.net/stats/cgi-bin/genstatspage.
>> fcgi?width=750&height=200&rrdfunc=MAX&rrdfile=%2Fvar%2Fflows%2Frrds%
>> 2FjnxBgpM2Prefixes%2Fdevice%3Dr-uwmadison-isp.wiscnet.net_
>> family%3Dipv4_class%3Dunicast_ip%3D4.53.34.5_
>> jnxBgpM2PrefixInPrefixesAccepted.rrd&time=CUSTOM&start_time=
>> 08%2F12%2F2014+2am&end_time=08%2F12%2F2014+6am
>>
>> fwiw,
>> -Michael
>>
>> On 8/12/2014 2:27 PM, Bill Wichers via Outages wrote:
>>
>>> I'm not so sure. None of my four upstreams is showing that many. Level3
>>> (as an example) is 497,389 right now.
>>>
>>> This might be an issue for some places but probably isn't the full "this
>>> is why everyone is down today" explanation.
>>>
>>>    -Bill
>>>
>>>
>>>  -----Original Message-----
>>>> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
>>>> Spencer Gaw via Outages
>>>> Sent: Tuesday, August 12, 2014 3:20 PM
>>>> To: Michael Peterman; outages at outages.org
>>>> Subject: Re: [outages] So what is broken
>>>>
>>>> I would imagine it's related to the global routing table hitting 512k
>>>> routes.
>>>>
>>>> http://www.cidr-report.org/cgi-
>>>> bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start
>>>> =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y
>>>> label=BGP+entries
>>>>
>>>> Regards,
>>>>
>>>> SG
>>>>
>>>> On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
>>>>
>>>>> So is this issue all related to a fiber cut or a DC/Peering point
>>>>> having issues?
>>>>>
>>>>> http://www.thewhir.com/web-hosting-news/liquidweb-among-
>>>>>
>>>> companies-affe
>>>>
>>>>> cted-major-outage-across-us-network-providers
>>>>>
>>>>>
>>>>>
>>>>> Michael  Peterman
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
>>>>> outages-request at outages.org
>>>>> Sent: Tuesday, August 12, 2014 11:00 AM
>>>>> To: outages at outages.org
>>>>> Subject: Outages Digest, Vol 75, Issue 9
>>>>>
>>>>> Send Outages mailing list submissions to
>>>>>         outages at outages.org
>>>>>
>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>         https://puck.nether.net/mailman/listinfo/outages
>>>>> or, via email, send a message with subject or body 'help' to
>>>>>         outages-request at outages.org
>>>>>
>>>>> You can reach the person managing the list at
>>>>>         outages-owner at outages.org
>>>>>
>>>>> When replying, please edit your Subject line so it is more specific
>>>>> than "Re:
>>>>>
>>>> Contents of Outages digest..."
>>>>
>>>>>
>>>>>
>>>>> Today's Topics:
>>>>>
>>>>>      1. Windstream (Mike Walter)
>>>>>      2. CoreSite Any2 LA (jd)
>>>>>      3. Re: CoreSite Any2 LA (Paul Stewart)
>>>>>      4. LastPass.com has a DC outage (Chris Egeland)
>>>>>      5. Major outages today, not much info at this time
>>>>>         (Hoyle Anderson (AM))
>>>>>      6. Re: Major outages today, not much info at this time (Teun Vink)
>>>>>
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>>
>>>>> Message: 1
>>>>> Date: Tue, 12 Aug 2014 13:19:08 +0000
>>>>> From: Mike Walter <mwalter at 3z.net>
>>>>> To: "'outages at outages.org'" <outages at outages.org>
>>>>> Subject: [outages] Windstream
>>>>> Message-ID:
>>>>>
>>>>>          <D7092CF606B8964193F248C92106B7810FCDE655 at zionex01.ZION.loc
>>>> al>
>>>>
>>>>> Content-Type: text/plain; charset="us-ascii"
>>>>>
>>>>> Just an FYI, there is a Windstream fiber cut effecting Ohio and
>>>>> Kentucky
>>>>>
>>>> customers, no ETA to repair based on their phone message.  Still on hold
>>>> waiting to open case to find out more.
>>>>
>>>>> -Mike
>>>>>
>>>>> -------------- next part -------------- An HTML attachment was
>>>>> scrubbed...
>>>>> URL:
>>>>>
>>>>>  <https://puck.nether.net/pipermail/outages/attachments/
>>>> 20140812/194c6c
>>>>
>>>>> 4b/attachment-0001.html>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Message: 2
>>>>> Date: Tue, 12 Aug 2014 07:19:04 -0700
>>>>> From: jd <jd at boldnetworks.net>
>>>>> To: "'outages at outages.org'" <outages at outages.org>
>>>>> Subject: [outages] CoreSite Any2 LA
>>>>> Message-ID: <53EA2258.2020601 at boldnetworks.net>
>>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>>
>>>>> Is anyone having issues with Any2 in LA? I have no evidence but I'm
>>>>> hearing
>>>>>
>>>> some things, if anyone could debunk.
>>>>
>>>>>
>>>>> Thanks
>>>>> JD
>>>>>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Message: 3
>>>>> Date: Tue, 12 Aug 2014 10:42:39 -0400
>>>>> From: Paul Stewart <paul at paulstewart.org>
>>>>> To: "'outages at outages.org'" <outages at outages.org>
>>>>> Subject: Re: [outages] CoreSite Any2 LA
>>>>> Message-ID: <D00FA00A.806C1%paul at paulstewart.org>
>>>>> Content-Type: text/plain;       charset="UTF-8"
>>>>>
>>>>> No issues here at this point?. At least that have become evident?
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>>> On 2014-08-12, 10:19 AM, "jd via Outages" <outages at outages.org> wrote:
>>>>>
>>>>>  Is anyone having issues with Any2 in LA? I have no evidence but I'm
>>>>>> hearing some things, if anyone could debunk.
>>>>>>
>>>>>> Thanks
>>>>>> JD
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Message: 4
>>>>> Date: Tue, 12 Aug 2014 15:04:58 +0000
>>>>> From: Chris Egeland <chris.egeland at ferrotherm.com>
>>>>> To: "outages at outages.org List" <outages at outages.org>
>>>>> Subject: [outages] LastPass.com has a DC outage
>>>>> Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB at mail01>
>>>>> Content-Type: text/plain; charset="us-ascii"
>>>>>
>>>>> https://twitter.com/LastPass/status/499199566701658112
>>>>>
>>>>> "Update: One of our data centers went down, we have the service fully
>>>>>
>>>> running on another data center & are working to minimize impact."
>>>>
>>>>>
>>>>> I'm getting "An error occurred while attempting to contact the server.
>>>>>
>>>> Please check your Internet connection." In my LastPass bar on Firefox.
>>>>
>>>>>
>>>>> So, if you're getting errors with LastPass, this is why.
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>>
>>>>> This e-mail and any attachments may contain privileged or confidential
>>>>>
>>>> information of Ferrotherm Corporation. This e-mail is intended solely
>>>> for the
>>>> use of the individual or entity to which it is addressed. If you are
>>>> not the
>>>> intended recipient of this e-mail, you are hereby notified that any
>>>> copying,
>>>> distribution, dissemination or action taken in relation to the contents
>>>> of this
>>>> e-mail and any of its attachments is strictly prohibited and may be
>>>> unlawful. If
>>>> you have received this e-mail in error, please notify the sender
>>>> immediately
>>>> and permanently delete the original e-mail and destroy any copies or
>>>> printouts of this e-mail as well as any attachments.
>>>>
>>>>> -------------- next part -------------- An HTML attachment was
>>>>> scrubbed...
>>>>> URL:
>>>>>
>>>>>  <https://puck.nether.net/pipermail/outages/attachments/
>>>> 20140812/837627
>>>>
>>>>> 0e/attachment-0001.html>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Message: 5
>>>>> Date: Tue, 12 Aug 2014 15:20:45 +0000
>>>>> From: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>>>>> To: "outages at outages.org" <outages at outages.org>
>>>>> Subject: [outages] Major outages today, not much info at this time
>>>>> Message-ID:
>>>>>
>>>>>
>>>>>  <7C6FA1D8224C7944A005E6587BDEA50A03F692 at usispsvexdb03.na.didata.lo
>>>> cal>
>>>>
>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>
>>>>> I know this isn?t much help, but there are major problems with multiple
>>>>>
>>>> ISPs since around 4-5 AM EST.  I really don?t have much detail, but I
>>>> have
>>>> sites that are unreachable from some providers.  Looks like Comcast,
>>>> level3,
>>>> ATT, cogent, etc.
>>>>
>>>>>
>>>>> So, it?s probably not just you, but I?m afraid I don?t know who it is.
>>>>>  I heard
>>>>>
>>>> one report of a datacenter outage.
>>>>
>>>>>
>>>>> Sources:
>>>>> http://downdetector.com/
>>>>> https://twitter.com/liquidweb
>>>>>
>>>>> -------------- next part -------------- An HTML attachment was
>>>>> scrubbed...
>>>>> URL:
>>>>>
>>>>>  <https://puck.nether.net/pipermail/outages/attachments/
>>>> 20140812/fa57b8
>>>>
>>>>> 50/attachment-0001.html>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Message: 6
>>>>> Date: Tue, 12 Aug 2014 17:42:05 +0200
>>>>> From: Teun Vink <teun at teun.tv>
>>>>> To: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>>>>> Cc: "outages at outages.org" <outages at outages.org>
>>>>> Subject: Re: [outages] Major outages today, not much info at this time
>>>>> Message-ID: <1407858125.27157.21.camel at moridin>
>>>>> Content-Type: text/plain; charset="UTF-8"
>>>>>
>>>>> On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages
>>>>> wrote:
>>>>>
>>>>>> I know this isn?t much help, but there are major problems with
>>>>>> multiple ISPs since around 4-5 AM EST.  I really don?t have much
>>>>>> detail, but I have sites that are unreachable from some providers.
>>>>>> Looks like Comcast, level3, ATT, cogent, etc.
>>>>>>
>>>>>>
>>>>>>
>>>>>> So, it?s probably not just you, but I?m afraid I don?t know who it is.
>>>>>> I heard one report of a datacenter outage.
>>>>>>
>>>>>>  Hi,
>>>>>
>>>>> Some routing tables hit 512K routes today. Some old hardware and
>>>>>
>>>> software can't handle that and either crash or ignore newly learned
>>>> routes.
>>>> So this may cause some disturbances in the force.
>>>>
>>>>>
>>>>> HTH,
>>>>> Teun
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Subject: Digest Footer
>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> End of Outages Digest, Vol 75, Issue 9
>>>>> **************************************
>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>  _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140812/6d2dbd14/attachment.htm>

More information about the Outages mailing list