[outages] Savvis/CenturyLink issues this AM

Tue Apr 22 22:09:58 EDT 2014

Yeah. This has been a disaster, and we've been seeing this since at least 9:00 AM CDT. I can't even make it from my apartment on the west side of Chicago over to CH3 without seeing 85% packet loss (via Comcast). 

The most recent we've heard is that they'll be doing their rebooting of stuff 6-8 hours after 3:00 PM, so 9:00 to 11:00. We'll see. In the interim, I think I'll order another pint. ;)

> On Apr 22, 2014, at 8:51 PM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> 
> It's convenient that there's no mention of when the "maintenances" will
> be (the DATE OF EVENT can't be it).  For those curious what this looks
> like (sorry I don't have return path traceroutes):
> 
> Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
> 1. gw.home.lan (192.168.1.1)                                        0.0%    31    31   0.2   0.3   0.2   0.4
> 2. 76.102.12.1                                                      0.0%    31    31   8.6   9.3   8.0  16.2
> 3. 68.86.249.253                                                    0.0%    31    31   8.3   9.2   8.2  16.7
> 4. te-1-1-0-12-ar01.oakland.ca.sfba.comcast.net (69.139.199.102)    0.0%    30    30  10.8  12.4  10.0  18.6
> 5. be-100-ar01.sfsutro.ca.sfba.comcast.net (68.85.155.18)           0.0%    30    30  10.8  12.3  10.3  14.0
> 6. he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.94.85)       0.0%    30    30  13.7  13.9  11.6  16.8
> 7. er1-tengig2-4.sanjoseequinix.savvis.net (206.24.222.1)           0.0%    30    30  12.1  13.6  11.9  29.9
> 8. er2-xe-7-0-0.SanJoseEquinix.savvis.net (204.70.198.142)          0.0%    30    30  56.1  16.6  11.7  56.1
> 9. cr2-tengig0-7-3-0.sanfrancisco.savvis.net (204.70.206.57)        0.0%    30    30  14.7  19.4  14.0  74.8
> 10. cr2-tengig-0-7-0-0.chicago.savvis.net (204.70.196.246)           3.3%    30    29  65.8  67.7  65.0  85.5
> 11. hr2-tengigabitethernet-12-1.elkgrovech3.savvis.net (204.70.195. 76.7%    30     7  66.1  66.9  65.3  72.7
> 12. das5-v3032.ch3.savvis.net (64.37.207.158)                       86.2%    30     4  69.0  80.1  65.2 120.8
> 13. 64.27.160.194                                                   82.8%    30     5  65.1  68.9  65.0  78.2
> 14. star.slashdot.org (216.34.181.48)                               72.4%    30     8  65.4  65.6  65.3  66.0
> 
> -- 
> | Jeremy Chadwick                                   jdc at koitsu.org |
> | UNIX Systems Administrator                http://jdc.koitsu.org/ |
> | Making life hard for others since 1977.             PGP 4BD6C0CB |
> 
>> On Tue, Apr 22, 2014 at 06:21:55PM -0400, Cary Wiedemann wrote:
>> This outage is still ongoing and has far-reaching implications beyond their
>> CH3 facility.  I have servers in DC3 (Sterling, VA/Washington, DC) and
>> haven't been able to reach 12.0.0.0/8 among other networks since early this
>> morning.
>> 
>> Looks like things will get worse before they get better.  Here's their
>> latest update:
>> 
>> DATE OF EVENT: Tuesday, April 22, 2014
>> TIME OF EVENT : 03:20 CDT
>> MASTER CASE: 4579123
>> 
>> UPDATE: In effort to resolve an ongoing incident, CenturyLink will be
>> reloading several internet facing backbone devices within our network.
>> This emergency maintenance will potentially result in an intermittent loss
>> of connectivity and/or latency for up to 15 minutes.  Clients with
>> non-redundant ATS MPLS and EVPL connectivity could experience additional
>> impact during this maintenance.  This activity is required to address a
>> resource constraint on several peering and hosting routers within the
>> CenturyLink backbone network.  CenturyLink will notify all impacted clients
>> via email at the beginning and after the completion of the maintenance
>> activity.  We apologize for the inconvenience.
>> 
>> CRC Management
>> Request at savvis.net
>> North America - 888 638 6771
>> EMEA - 00800 7288 4743
>> Asia-Pacific - +65 63058099
>> 
>> - Cary Wiedemann
>> carywiedemann at gmail.com
>> +1 703 592 6498
>> 
>> 
>> On Tue, Apr 22, 2014 at 10:34 AM, Bill Weiss
>> <houdini+outages at clanspum.net>wrote:
>> 
>>> Unsure if this is related to the L3 issues or not.  Savvis, at least at
>>> the CH3 datacenter, has had bad connectivity problems since 3:20AM CDT.
>>> Internet Pulse shows atrocious packet loss to almost everywhere from
>>> Savvis.
>>> 
>>> Master case number at Savvis is 4579123.  They don't have any more info
>>> than "yep, we're down, working on it".
>>> 
>>> --
>>> Bill Weiss
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
> 
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages