[outages] So what is broken

Tue Aug 12 17:29:04 EDT 2014

We were told by Quality Technology Services in Atlanta that they were "installing additional memory" to correct the issue.  Perhaps they meant they were actually reconfiguring/bouncing the routers.... I know they had Cisco engineers on site in GA, CA, and NJ today.

Either way, after our network link went totally down and came back up, the issue was resolved.

________________________________
From: Bill Wichers [billw at waveform.net]
Sent: Tuesday, August 12, 2014 3:27 PM
To: Geoffrey Mina
Cc: Michael Peterman; outages at outages.org
Subject: Re: [outages] So what is broken

That's a tcam limit on the 6500/7600 platform. It's not upgradable, but it is configurable.

The gsr platform doesn't have a hard limit like this, its just memory (And is upgradeable), but most with that gear will have already had to upgrade well before now.

Sent from my iPhone

On Aug 12, 2014, at 4:14 PM, "Geoffrey Mina via Outages" <outages at outages.org<mailto:outages at outages.org>> wrote:

Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today.

ISPs are scrambling to update memory in their routers.

-------- Original message --------
From: Michael Peterman via Outages
Date:08/12/2014 1:00 PM (GMT-07:00)
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] So what is broken

So is this issue all related to a fiber cut or a DC/Peering point having issues?

http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers

Michael  Peterman

-----Original Message-----
From: Outages [mailto:outages-bounces at outages.org] On Behalf Of outages-request at outages.org<mailto:outages-request at outages.org>
Sent: Tuesday, August 12, 2014 11:00 AM
To: outages at outages.org<mailto:outages at outages.org>
Subject: Outages Digest, Vol 75, Issue 9

Send Outages mailing list submissions to
        outages at outages.org<mailto:outages at outages.org>

To subscribe or unsubscribe via the World Wide Web, visit
        https://puck.nether.net/mailman/listinfo/outages
or, via email, send a message with subject or body 'help' to
        outages-request at outages.org<mailto:outages-request at outages.org>

You can reach the person managing the list at
        outages-owner at outages.org<mailto:outages-owner at outages.org>

When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."

Today's Topics:

   1. Windstream (Mike Walter)
   2. CoreSite Any2 LA (jd)
   3. Re: CoreSite Any2 LA (Paul Stewart)
   4. LastPass.com<http://LastPass.com> has a DC outage (Chris Egeland)
   5. Major outages today, not much info at this time
      (Hoyle Anderson (AM))
   6. Re: Major outages today, not much info at this time (Teun Vink)

----------------------------------------------------------------------

Message: 1
Date: Tue, 12 Aug 2014 13:19:08 +0000
From: Mike Walter <mwalter at 3z.net<mailto:mwalter at 3z.net>>
To: "'outages at outages.org<mailto:outages at outages.org>'" <outages at outages.org<mailto:outages at outages.org>>
Subject: [outages] Windstream
Message-ID:
        <D7092CF606B8964193F248C92106B7810FCDE655 at zionex01.ZION.local<mailto:D7092CF606B8964193F248C92106B7810FCDE655 at zionex01.ZION.local>>
Content-Type: text/plain; charset="us-ascii"

Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message.  Still on hold waiting to open case to find out more.
-Mike

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c4b/attachment-0001.html>

------------------------------

Message: 2
Date: Tue, 12 Aug 2014 07:19:04 -0700
From: jd <jd at boldnetworks.net<mailto:jd at boldnetworks.net>>
To: "'outages at outages.org<mailto:outages at outages.org>'" <outages at outages.org<mailto:outages at outages.org>>
Subject: [outages] CoreSite Any2 LA
Message-ID: <53EA2258.2020601 at boldnetworks.net<mailto:53EA2258.2020601 at boldnetworks.net>>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.

Thanks
JD

------------------------------

Message: 3
Date: Tue, 12 Aug 2014 10:42:39 -0400
From: Paul Stewart <paul at paulstewart.org<mailto:paul at paulstewart.org>>
To: "'outages at outages.org<mailto:outages at outages.org>'" <outages at outages.org<mailto:outages at outages.org>>
Subject: Re: [outages] CoreSite Any2 LA
Message-ID: <D00FA00A.806C1%paul at paulstewart.org<mailto:paul at paulstewart.org>>
Content-Type: text/plain;       charset="UTF-8"

No issues here at this point?. At least that have become evident?

Paul

On 2014-08-12, 10:19 AM, "jd via Outages" <outages at outages.org<mailto:outages at outages.org>> wrote:

>Is anyone having issues with Any2 in LA? I have no evidence but I'm
>hearing some things, if anyone could debunk.
>
>Thanks
>JD
>_______________________________________________
>Outages mailing list
>Outages at outages.org<mailto:Outages at outages.org>
>https://puck.nether.net/mailman/listinfo/outages
>

------------------------------

Message: 4
Date: Tue, 12 Aug 2014 15:04:58 +0000
From: Chris Egeland <chris.egeland at ferrotherm.com<mailto:chris.egeland at ferrotherm.com>>
To: "outages at outages.org<mailto:outages at outages.org> List" <outages at outages.org<mailto:outages at outages.org>>
Subject: [outages] LastPass.com<http://LastPass.com> has a DC outage
Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB at mail01>
Content-Type: text/plain; charset="us-ascii"

https://twitter.com/LastPass/status/499199566701658112

"Update: One of our data centers went down, we have the service fully running on another data center & are working to minimize impact."

I'm getting "An error occurred while attempting to contact the server. Please check your Internet connection." In my LastPass bar on Firefox.

So, if you're getting errors with LastPass, this is why.

________________________________

This e-mail and any attachments may contain privileged or confidential information of Ferrotherm Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any copying, distribution, dissemination or action taken in relation to the contents of this e-mail and any of its attachments is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original e-mail and destroy any copies or printouts of this e-mail as well as any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140812/8376270e/attachment-0001.html>

------------------------------

Message: 5
Date: Tue, 12 Aug 2014 15:20:45 +0000
From: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com<mailto:hoyle.anderson at dimensiondata.com>>
To: "outages at outages.org<mailto:outages at outages.org>" <outages at outages.org<mailto:outages at outages.org>>
Subject: [outages] Major outages today, not much info at this time
Message-ID:
        <7C6FA1D8224C7944A005E6587BDEA50A03F692 at usispsvexdb03.na.didata.local<mailto:7C6FA1D8224C7944A005E6587BDEA50A03F692 at usispsvexdb03.na.didata.local>>
Content-Type: text/plain; charset="utf-8"

I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST.  I really don?t have much detail, but I have sites that are unreachable from some providers.  Looks like Comcast, level3, ATT, cogent, etc.

So, it?s probably not just you, but I?m afraid I don?t know who it is.  I heard one report of a datacenter outage.

Sources:
http://downdetector.com/
https://twitter.com/liquidweb

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b850/attachment-0001.html>

------------------------------

Message: 6
Date: Tue, 12 Aug 2014 17:42:05 +0200
From: Teun Vink <teun at teun.tv<mailto:teun at teun.tv>>
To: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com<mailto:hoyle.anderson at dimensiondata.com>>
Cc: "outages at outages.org<mailto:outages at outages.org>" <outages at outages.org<mailto:outages at outages.org>>
Subject: Re: [outages] Major outages today, not much info at this time
Message-ID: <1407858125.27157.21.camel at moridin>
Content-Type: text/plain; charset="UTF-8"

On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
> I know this isn?t much help, but there are major problems with
> multiple ISPs since around 4-5 AM EST.  I really don?t have much
> detail, but I have sites that are unreachable from some providers.
> Looks like Comcast, level3, ATT, cogent, etc.
>
>
>
> So, it?s probably not just you, but I?m afraid I don?t know who it is.
> I heard one report of a datacenter outage.
>

Hi,

Some routing tables hit 512K routes today. Some old hardware and software can't handle that and either crash or ignore newly learned routes. So this may cause some disturbances in the force.

HTH,
Teun

------------------------------

Subject: Digest Footer

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

------------------------------

End of Outages Digest, Vol 75, Issue 9
**************************************

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140812/9c894654/attachment.htm>