[outages] So what is broken

Wed Aug 13 03:12:40 EDT 2014

Looks like "full route" routing tables peaked at ~515k routes briefly
due to an aggregation issue at Verizon. We saw the increase in several
of our full table feeds from Level3 today.

More details here: http://www.bgpmon.net/what-caused-todays-internet-hiccup/

Cheers,
 Andree

.-- My secret spy satellite informs me that at 2014-08-12 1:38 PM
Christopher L. Stone via Outages wrote:
> Only seeing 497, 554 IPv4 routes and 18,128 IPv6 routes here as of right
> now through L3.
> 
> 
> Chris
> 
> On 08/12/2014 01:44 PM, Geoffrey Mina via Outages wrote:
>> Cisco gear can't support more than 512K routes... We hit that or close
>> to that at around 4am EDT today. 
>>
>> ISPs are scrambling to update memory in their routers.
>>
>>
>> -------- Original message --------
>> From: Michael Peterman via Outages
>> Date:08/12/2014 1:00 PM (GMT-07:00)
>> To: outages at outages.org
>> Subject: [outages] So what is broken
>>
>> So is this issue all related to a fiber cut or a DC/Peering point
>> having issues?
>>
>> http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers
>>
>>
>>
>> Michael  Peterman
>>
>>
>>
>>
>> -----Original Message-----
>> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
>> outages-request at outages.org
>> Sent: Tuesday, August 12, 2014 11:00 AM
>> To: outages at outages.org
>> Subject: Outages Digest, Vol 75, Issue 9
>>
>> Send Outages mailing list submissions to
>>         outages at outages.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://puck.nether.net/mailman/listinfo/outages
>> or, via email, send a message with subject or body 'help' to
>>         outages-request at outages.org
>>
>> You can reach the person managing the list at
>>         outages-owner at outages.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Outages digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Windstream (Mike Walter)
>>    2. CoreSite Any2 LA (jd)
>>    3. Re: CoreSite Any2 LA (Paul Stewart)
>>    4. LastPass.com has a DC outage (Chris Egeland)
>>    5. Major outages today, not much info at this time
>>       (Hoyle Anderson (AM))
>>    6. Re: Major outages today, not much info at this time (Teun Vink)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 12 Aug 2014 13:19:08 +0000
>> From: Mike Walter <mwalter at 3z.net>
>> To: "'outages at outages.org'" <outages at outages.org>
>> Subject: [outages] Windstream
>> Message-ID:
>>         <D7092CF606B8964193F248C92106B7810FCDE655 at zionex01.ZION.local>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Just an FYI, there is a Windstream fiber cut effecting Ohio and
>> Kentucky customers, no ETA to repair based on their phone message. 
>> Still on hold waiting to open case to find out more.
>> -Mike
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c4b/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 12 Aug 2014 07:19:04 -0700
>> From: jd <jd at boldnetworks.net>
>> To: "'outages at outages.org'" <outages at outages.org>
>> Subject: [outages] CoreSite Any2 LA
>> Message-ID: <53EA2258.2020601 at boldnetworks.net>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Is anyone having issues with Any2 in LA? I have no evidence but I'm
>> hearing some things, if anyone could debunk.
>>
>> Thanks
>> JD
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Tue, 12 Aug 2014 10:42:39 -0400
>> From: Paul Stewart <paul at paulstewart.org>
>> To: "'outages at outages.org'" <outages at outages.org>
>> Subject: Re: [outages] CoreSite Any2 LA
>> Message-ID: <D00FA00A.806C1%paul at paulstewart.org>
>> Content-Type: text/plain;       charset="UTF-8"
>>
>> No issues here at this point?. At least that have become evident?
>>
>> Paul
>>
>>
>>
>> On 2014-08-12, 10:19 AM, "jd via Outages" <outages at outages.org> wrote:
>>
>> >Is anyone having issues with Any2 in LA? I have no evidence but I'm
>> >hearing some things, if anyone could debunk.
>> >
>> >Thanks
>> >JD
>> >_______________________________________________
>> >Outages mailing list
>> >Outages at outages.org
>> >https://puck.nether.net/mailman/listinfo/outages
>> >
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Tue, 12 Aug 2014 15:04:58 +0000
>> From: Chris Egeland <chris.egeland at ferrotherm.com>
>> To: "outages at outages.org List" <outages at outages.org>
>> Subject: [outages] LastPass.com has a DC outage
>> Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB at mail01>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> https://twitter.com/LastPass/status/499199566701658112
>>
>> "Update: One of our data centers went down, we have the service fully
>> running on another data center & are working to minimize impact."
>>
>> I'm getting "An error occurred while attempting to contact the server.
>> Please check your Internet connection." In my LastPass bar on Firefox.
>>
>> So, if you're getting errors with LastPass, this is why.
>>
>>
>>
>> ________________________________
>>
>> This e-mail and any attachments may contain privileged or confidential
>> information of Ferrotherm Corporation. This e-mail is intended solely
>> for the use of the individual or entity to which it is addressed. If
>> you are not the intended recipient of this e-mail, you are hereby
>> notified that any copying, distribution, dissemination or action taken
>> in relation to the contents of this e-mail and any of its attachments
>> is strictly prohibited and may be unlawful. If you have received this
>> e-mail in error, please notify the sender immediately and permanently
>> delete the original e-mail and destroy any copies or printouts of this
>> e-mail as well as any attachments.
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <https://puck.nether.net/pipermail/outages/attachments/20140812/8376270e/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Tue, 12 Aug 2014 15:20:45 +0000
>> From: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>> To: "outages at outages.org" <outages at outages.org>
>> Subject: [outages] Major outages today, not much info at this time
>> Message-ID:
>>        
>> <7C6FA1D8224C7944A005E6587BDEA50A03F692 at usispsvexdb03.na.didata.local>
>> Content-Type: text/plain; charset="utf-8"
>>
>> I know this isn?t much help, but there are major problems with
>> multiple ISPs since around 4-5 AM EST.  I really don?t have much
>> detail, but I have sites that are unreachable from some providers. 
>> Looks like Comcast, level3, ATT, cogent, etc.
>>
>> So, it?s probably not just you, but I?m afraid I don?t know who it
>> is.  I heard one report of a datacenter outage.
>>
>> Sources:
>> http://downdetector.com/
>> https://twitter.com/liquidweb
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b850/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 6
>> Date: Tue, 12 Aug 2014 17:42:05 +0200
>> From: Teun Vink <teun at teun.tv>
>> To: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
>> Cc: "outages at outages.org" <outages at outages.org>
>> Subject: Re: [outages] Major outages today, not much info at this time
>> Message-ID: <1407858125.27157.21.camel at moridin>
>> Content-Type: text/plain; charset="UTF-8"
>>
>> On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
>> > I know this isn?t much help, but there are major problems with
>> > multiple ISPs since around 4-5 AM EST.  I really don?t have much
>> > detail, but I have sites that are unreachable from some providers.
>> > Looks like Comcast, level3, ATT, cogent, etc.
>> >
>> > 
>> >
>> > So, it?s probably not just you, but I?m afraid I don?t know who it is.
>> > I heard one report of a datacenter outage.
>> >
>>
>> Hi,
>>
>> Some routing tables hit 512K routes today. Some old hardware and
>> software can't handle that and either crash or ignore newly learned
>> routes. So this may cause some disturbances in the force.
>>
>> HTH,
>> Teun
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> ------------------------------
>>
>> End of Outages Digest, Vol 75, Issue 9
>> **************************************
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages