[outages] So what is broken

Sean Hayes sean at engage.pro
Tue Aug 12 16:12:19 EDT 2014 

CIDR report is showing over 512k around the same time.
On Aug 12, 2014 4:09 PM, "Michael Hare via Outages" <outages at outages.org>
wrote:

> AS 2381 definitely saw Level3 sending > 512K for a brief time this AM, say
> 02:50 to 03:25.  Perhaps those affected haven't figured it out yet.  My
> below graph has timestamps in CDT.
>
> https://www.eng.wiscnet.net/stats/cgi-bin/genstatspage.
> fcgi?width=750&height=200&rrdfunc=MAX&rrdfile=%2Fvar%2Fflows%2Frrds%
> 2FjnxBgpM2Prefixes%2Fdevice%3Dr-uwmadison-isp.wiscnet.net_
> family%3Dipv4_class%3Dunicast_ip%3D4.53.34.5_
> jnxBgpM2PrefixInPrefixesAccepted.rrd&time=CUSTOM&start_time=
> 08%2F12%2F2014+2am&end_time=08%2F12%2F2014+6am
>
> fwiw,
> -Michael
>
> On 8/12/2014 2:27 PM, Bill Wichers via Outages wrote:
>
>> I'm not so sure. None of my four upstreams is showing that many. Level3
>> (as an example) is 497,389 right now.
>>
>> This might be an issue for some places but probably isn't the full "this
>> is why everyone is down today" explanation.
>>
>>    -Bill
>>
>>
>>  -----Original Message-----
>>> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
>>> Spencer Gaw via Outages
>>> Sent: Tuesday, August 12, 2014 3:20 PM
>>> To: Michael Peterman; outages at outages.org
>>> Subject: Re: [outages] So what is broken
>>>
>>> I would imagine it's related to the global routing table hitting 512k
>>> routes.
>>>
>>> http://www.cidr-report.org/cgi-
>>> bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start
>>> =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y
>>> label=BGP+entries
>>>
>>> Regards,
>>>
>>> SG
>>>
>>> On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
>>>
>>>> So is this issue all related to a fiber cut or a DC/Peering point
>>>> having issues?
>>>>
>>>> http://www.thewhir.com/web-hosting-news/liquidweb-among-
>>>>
>>> companies-affe
>>>
>>>> cted-major-outage-across-us-network-providers
>>>>
>>>>
>>>>
>>>> Michael  Peterman
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
>>>> outages-request at outages.org
>>>> Sent: Tuesday, August 12, 2014 11:00 AM
>>>> To: outages at outages.org
>>>> Subject: Outages Digest, Vol 75, Issue 9
>>>>
>>>> Send Outages mailing list submissions to
>>>>         outages at outages.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>         https://puck.nether.net/mailman/listinfo/outages
>>>> or, via email, send a message with subject or body 'help' to
>>>>         outages-request at outages.org
>>>>
>>>> You can reach the person managing the list at
>>>>         outages-owner at outages.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re:
>>>>
>>> Contents of Outages digest..."
>>>
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>      1. Windstream (Mike Walter)
>>>>      2. CoreSite Any2 LA (jd)
>>>>      3. Re: CoreSite Any2 LA (Paul Stewart)
>>>>      4. LastPass.com has a DC outage (Chris Egeland)
>>>>      5. Major outages today, not much info at this time
>>>>         (Hoyle Anderson (AM))
>>>>      6. Re: Major outages today, not much info at this time (Teun Vink)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Tue, 12 Aug 2014 13:19:08 +0000
>>>> From: Mike Walter <mwalter at 3z.net>
>>>> To: "'outages at outages.org'" <outages at outages.org>
>>>> Subject: [outages] Windstream
>>>> Message-ID:
>>>>
>>>>          <D7092CF606B8964193F248C92106B7810FCDE655 at zionex01.ZION.loc
>>> al>
>>>
>>>> Content-Type: text/plain; charset="us-ascii"
>>>>
>>>> Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky
>>>>
>>> customers, no ETA to repair based on their phone message.  Still on hold
>>> waiting to open case to find out more.
>>>
>>>> -Mike
>>>>
>>>> -------------- next part -------------- An HTML attachment was
>>>> scrubbed...
>>>> URL:
>>>>
>>>>  <https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c
>>>
>>>> 4b/attachment-0001.html>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 2
>>>> Date: Tue, 12 Aug 2014 07:19:04 -0700
>>>> From: jd <jd at boldnetworks.net>
>>>> To: "'outages at outages.org'" <outages at outages.org>
>>>> Subject: [outages] CoreSite Any2 LA
>>>> Message-ID: <53EA2258.2020601 at boldnetworks.net>
>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>
>>>> Is anyone having issues with Any2 in LA? I have no evidence but I'm
>>>> hearing
>>>>
>>> some things, if anyone could debunk.
>>>
>>>>
>>>> Thanks
>>>> JD
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 3
>>>> Date: Tue, 12 Aug 2014 10:42:39 -0400
>>>> From: Paul Stewart <paul at paulstewart.org>
>>>> To: "'outages at outages.org'" <outages at outages.org>
>>>> Subject: Re: [outages] CoreSite Any2 LA
>>>> Message-ID: <D00FA00A.806C1%paul at paulstewart.org>
>>>> Content-Type: text/plain;       charset="UTF-8"
>>>>
>>>> No issues here at this point?. At least that have become evident?
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>> On 2014-08-12, 10:19 AM, "jd via Outages" <outages at outages.org> wrote:
>>>>
>>>>  Is anyone having issues with Any2 in LA? I have no evidence but I'm
>>>>> hearing some things, if anyone could debunk.
>>>>>
>>>>> Thanks
>>>>> JD
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 4
>>>> Date: Tue, 12 Aug 2014 15:04:58 +0000
>>>> From: Chris Egeland <chris.egeland at ferrotherm.com>
>>>> To: "outages at outages.org List" <outages at outages.org>
>>>> Subject: [outages] LastPass.com has a DC outage
>>>> Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB at mail01>
>>>> Content-Type: text/plain; charset="us-ascii"
>>>>
>>>> https://twitter.com/LastPass/status/499199566701658112
>>>>
>>>> "Update: One of our data centers went down, we have the service fully
>>>>
>>> running on another data center & are working to minimize impact."
>>>
>>>>
>>>> I'm getting "An error occurred while attempting to contact the server.
>>>>
>>> Please check your Internet connection." In my LastPass bar on Firefox.
>>>
>>>>
>>>> So, if you're getting errors with LastPass, this is why.
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> This e-mail and any attachments may contain privileged or confidential
>>>>
>>> information of Ferrotherm Corporation. This e-mail is intended solely
>>> for the
>>> use of the individual or entity to which it is addressed. If you are not
>>> the
>>> intended recipient of this e-mail, you are hereby notified that any
>>> copying,
>>> distribution, dissemination or action taken in relation to the contents
>>> of this
>>> e-mail and any of its attachments is strictly prohibited and may be
>>> unlawful. If
>>> you have received this e-mail in error, please notify the sender
>>> immediately
>>> and permanently delete the original e-mail and destroy any copies or
>>> printouts of this e-mail as well as any attachments.
>>>
>>>> -------------- next part -------------- An HTML attachment was
>>>> scrubbed...
>>>> URL:
>>>>
>>>>  <https://puck.nether.net/pipermail/outages/attachments/20140812/837627
>>>
>>>> 0e/attachment-0001.html>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 5
>>>> Date: Tue, 12 Aug 2014 15:20:45 +0000
>>>> From: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>>>> To: "outages at outages.org" <outages at outages.org>
>>>> Subject: [outages] Major outages today, not much info at this time
>>>> Message-ID:
>>>>
>>>>
>>>>  <7C6FA1D8224C7944A005E6587BDEA50A03F692 at usispsvexdb03.na.didata.lo
>>> cal>
>>>
>>>> Content-Type: text/plain; charset="utf-8"
>>>>
>>>> I know this isn?t much help, but there are major problems with multiple
>>>>
>>> ISPs since around 4-5 AM EST.  I really don?t have much detail, but I
>>> have
>>> sites that are unreachable from some providers.  Looks like Comcast,
>>> level3,
>>> ATT, cogent, etc.
>>>
>>>>
>>>> So, it?s probably not just you, but I?m afraid I don?t know who it is.
>>>>  I heard
>>>>
>>> one report of a datacenter outage.
>>>
>>>>
>>>> Sources:
>>>> http://downdetector.com/
>>>> https://twitter.com/liquidweb
>>>>
>>>> -------------- next part -------------- An HTML attachment was
>>>> scrubbed...
>>>> URL:
>>>>
>>>>  <https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8
>>>
>>>> 50/attachment-0001.html>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 6
>>>> Date: Tue, 12 Aug 2014 17:42:05 +0200
>>>> From: Teun Vink <teun at teun.tv>
>>>> To: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>>>> Cc: "outages at outages.org" <outages at outages.org>
>>>> Subject: Re: [outages] Major outages today, not much info at this time
>>>> Message-ID: <1407858125.27157.21.camel at moridin>
>>>> Content-Type: text/plain; charset="UTF-8"
>>>>
>>>> On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
>>>>
>>>>> I know this isn?t much help, but there are major problems with
>>>>> multiple ISPs since around 4-5 AM EST.  I really don?t have much
>>>>> detail, but I have sites that are unreachable from some providers.
>>>>> Looks like Comcast, level3, ATT, cogent, etc.
>>>>>
>>>>>
>>>>>
>>>>> So, it?s probably not just you, but I?m afraid I don?t know who it is.
>>>>> I heard one report of a datacenter outage.
>>>>>
>>>>>  Hi,
>>>>
>>>> Some routing tables hit 512K routes today. Some old hardware and
>>>>
>>> software can't handle that and either crash or ignore newly learned
>>> routes.
>>> So this may cause some disturbances in the force.
>>>
>>>>
>>>> HTH,
>>>> Teun
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Subject: Digest Footer
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> End of Outages Digest, Vol 75, Issue 9
>>>> **************************************
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>  _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140812/bfb7d1f9/attachment.htm>

More information about the Outages mailing list