[outages] So what is broken

Warren Kumari warren at kumari.net
Tue Aug 12 17:59:18 EDT 2014 

On Tue, Aug 12, 2014 at 5:29 PM, Geoffrey Mina via Outages
<outages at outages.org> wrote:
> We were told by Quality Technology Services in Atlanta that they were
> "installing additional memory" to correct the issue.  Perhaps they meant
> they were actually reconfiguring/bouncing the routers.... I know they had
> Cisco engineers on site in GA, CA, and NJ today.
>

Probably they were recarving the TCAM allocation.
Quick summary document for those having the issue:
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/117712-problemsolution-cat6500-00.html

This increases the size of the TCAM allocated for v4 routes, at the
cost of v6 space. This doesn't actually *solve* the issue, but does at
least kick the can down the road a bit longer...
TCAM based solutions for FIB, while cheap to implement, leads to pain...

W


> Either way, after our network link went totally down and came back up, the
> issue was resolved.
>
>
> ________________________________
> From: Bill Wichers [billw at waveform.net]
> Sent: Tuesday, August 12, 2014 3:27 PM
> To: Geoffrey Mina
> Cc: Michael Peterman; outages at outages.org
> Subject: Re: [outages] So what is broken
>
> That's a tcam limit on the 6500/7600 platform. It's not upgradable, but it
> is configurable.
>
> The gsr platform doesn't have a hard limit like this, its just memory (And
> is upgradeable), but most with that gear will have already had to upgrade
> well before now.
>
> Sent from my iPhone
>
> On Aug 12, 2014, at 4:14 PM, "Geoffrey Mina via Outages"
> <outages at outages.org> wrote:
>
> Cisco gear can't support more than 512K routes... We hit that or close to
> that at around 4am EDT today.
>
> ISPs are scrambling to update memory in their routers.
>
>
> -------- Original message --------
> From: Michael Peterman via Outages
> Date:08/12/2014 1:00 PM (GMT-07:00)
> To: outages at outages.org
> Subject: [outages] So what is broken
>
> So is this issue all related to a fiber cut or a DC/Peering point having
> issues?
>
> http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers
>
>
>
> Michael  Peterman
>
>
>
>
> -----Original Message-----
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
> outages-request at outages.org
> Sent: Tuesday, August 12, 2014 11:00 AM
> To: outages at outages.org
> Subject: Outages Digest, Vol 75, Issue 9
>
> Send Outages mailing list submissions to
>         outages at outages.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://puck.nether.net/mailman/listinfo/outages
> or, via email, send a message with subject or body 'help' to
>         outages-request at outages.org
>
> You can reach the person managing the list at
>         outages-owner at outages.org
>
> When replying, please edit your Subject line so it is more specific than
> "Re: Contents of Outages digest..."
>
>
> Today's Topics:
>
>    1. Windstream (Mike Walter)
>    2. CoreSite Any2 LA (jd)
>    3. Re: CoreSite Any2 LA (Paul Stewart)
>    4. LastPass.com has a DC outage (Chris Egeland)
>    5. Major outages today, not much info at this time
>       (Hoyle Anderson (AM))
>    6. Re: Major outages today, not much info at this time (Teun Vink)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 12 Aug 2014 13:19:08 +0000
> From: Mike Walter <mwalter at 3z.net>
> To: "'outages at outages.org'" <outages at outages.org>
> Subject: [outages] Windstream
> Message-ID:
>         <D7092CF606B8964193F248C92106B7810FCDE655 at zionex01.ZION.local>
> Content-Type: text/plain; charset="us-ascii"
>
> Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky
> customers, no ETA to repair based on their phone message.  Still on hold
> waiting to open case to find out more.
> -Mike
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c4b/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 12 Aug 2014 07:19:04 -0700
> From: jd <jd at boldnetworks.net>
> To: "'outages at outages.org'" <outages at outages.org>
> Subject: [outages] CoreSite Any2 LA
> Message-ID: <53EA2258.2020601 at boldnetworks.net>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing
> some things, if anyone could debunk.
>
> Thanks
> JD
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 12 Aug 2014 10:42:39 -0400
> From: Paul Stewart <paul at paulstewart.org>
> To: "'outages at outages.org'" <outages at outages.org>
> Subject: Re: [outages] CoreSite Any2 LA
> Message-ID: <D00FA00A.806C1%paul at paulstewart.org>
> Content-Type: text/plain;       charset="UTF-8"
>
> No issues here at this point?. At least that have become evident?
>
> Paul
>
>
>
> On 2014-08-12, 10:19 AM, "jd via Outages" <outages at outages.org> wrote:
>
>>Is anyone having issues with Any2 in LA? I have no evidence but I'm
>>hearing some things, if anyone could debunk.
>>
>>Thanks
>>JD
>>_______________________________________________
>>Outages mailing list
>>Outages at outages.org
>>https://puck.nether.net/mailman/listinfo/outages
>>
>
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 12 Aug 2014 15:04:58 +0000
> From: Chris Egeland <chris.egeland at ferrotherm.com>
> To: "outages at outages.org List" <outages at outages.org>
> Subject: [outages] LastPass.com has a DC outage
> Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB at mail01>
> Content-Type: text/plain; charset="us-ascii"
>
> https://twitter.com/LastPass/status/499199566701658112
>
> "Update: One of our data centers went down, we have the service fully
> running on another data center & are working to minimize impact."
>
> I'm getting "An error occurred while attempting to contact the server.
> Please check your Internet connection." In my LastPass bar on Firefox.
>
> So, if you're getting errors with LastPass, this is why.
>
>
>
> ________________________________
>
> This e-mail and any attachments may contain privileged or confidential
> information of Ferrotherm Corporation. This e-mail is intended solely for
> the use of the individual or entity to which it is addressed. If you are not
> the intended recipient of this e-mail, you are hereby notified that any
> copying, distribution, dissemination or action taken in relation to the
> contents of this e-mail and any of its attachments is strictly prohibited
> and may be unlawful. If you have received this e-mail in error, please
> notify the sender immediately and permanently delete the original e-mail and
> destroy any copies or printouts of this e-mail as well as any attachments.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://puck.nether.net/pipermail/outages/attachments/20140812/8376270e/attachment-0001.html>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 12 Aug 2014 15:20:45 +0000
> From: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
> To: "outages at outages.org" <outages at outages.org>
> Subject: [outages] Major outages today, not much info at this time
> Message-ID:
>
> <7C6FA1D8224C7944A005E6587BDEA50A03F692 at usispsvexdb03.na.didata.local>
> Content-Type: text/plain; charset="utf-8"
>
> I know this isn?t much help, but there are major problems with multiple ISPs
> since around 4-5 AM EST.  I really don?t have much detail, but I have sites
> that are unreachable from some providers.  Looks like Comcast, level3, ATT,
> cogent, etc.
>
> So, it?s probably not just you, but I?m afraid I don?t know who it is.  I
> heard one report of a datacenter outage.
>
> Sources:
> http://downdetector.com/
> https://twitter.com/liquidweb
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b850/attachment-0001.html>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 12 Aug 2014 17:42:05 +0200
> From: Teun Vink <teun at teun.tv>
> To: "Hoyle Anderson (AM)" <hoyle.anderson at dimensiondata.com>
> Cc: "outages at outages.org" <outages at outages.org>
> Subject: Re: [outages] Major outages today, not much info at this time
> Message-ID: <1407858125.27157.21.camel at moridin>
> Content-Type: text/plain; charset="UTF-8"
>
> On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
>> I know this isn?t much help, but there are major problems with
>> multiple ISPs since around 4-5 AM EST.  I really don?t have much
>> detail, but I have sites that are unreachable from some providers.
>> Looks like Comcast, level3, ATT, cogent, etc.
>>
>>
>>
>> So, it?s probably not just you, but I?m afraid I don?t know who it is.
>> I heard one report of a datacenter outage.
>>
>
> Hi,
>
> Some routing tables hit 512K routes today. Some old hardware and software
> can't handle that and either crash or ignore newly learned routes. So this
> may cause some disturbances in the force.
>
> HTH,
> Teun
>
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ------------------------------
>
> End of Outages Digest, Vol 75, Issue 9
> **************************************
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

More information about the Outages mailing list