[outages] XO Outage

[blackham at gmail.com]

Finally got the RFO:

Summary: On Monday September 22, 2008 at 4:12 PM all XO Peering Routers were isolated due to a mis-configuration caused by human error.  An XO technician inadvertently ran an incomplete script while attempting to complete non-impacting changes on an SNMP filter.  XO data customers were unable to reach external websites following the scripting push.

XO identified the issue immediately and began the process of logging into the Peering routers and corrected the configuration to restore the Peering router connectivity by 5:20 PM.

As of this writing, XO operations is implementing a verification screen prior to script launching to ensure that only the intended scripts are being deployed. In addition, all scripting will require a Scheduled Maintenance Request or Repair event.

Cause: Inadvertent mis-configuration caused by human error
Resolution: XO technicians corrected configuration to restore peering router connectivity.
Sent from my BlackBerry® smartphone with SprintSpeed

-----Original Message-----
From: blackham at gmail.com

Date: Tue, 23 Sep 2008 00:41:42 
To: Paul Norton<paul at neoverve.com>; <outages-bounces at outages.org>; Justin Sharp<sharp at sharpone.net>
Cc: <outages at outages.org>
Subject: Re: [outages] XO Outage


Little late on my part, but yes... XO partitioned for 20-30 mins today. Said it was a peering issue. I'll share the RFO when I get it.  
Sent from my BlackBerry® smartphone with SprintSpeed

-----Original Message-----
From: Paul Norton <paul at neoverve.com>

Date: Mon, 22 Sep 2008 14:53:43 
To: Justin Sharp<sharp at sharpone.net>
Cc: <outages at outages.org>
Subject: Re: [outages] XO Outage


Seeing outage here in San Diego.

On Sep 22, 2008, at 2:35 PM, Justin Sharp wrote:

> We are seeing some issues w/ XO/Savvis peering..
>
> Trace from XO to Savvis IP space (64.75.10.151)
>
> Keys:  Help   Display mode   Restart statistics   Order of fields    
> quit
>                                                                                                        Packets 
>                Pings
> Host 
>                                                                                                   Loss 
> %   Snt   Last   Avg  Best  Wrst StDev
> 1.  
> scrubbed 
>                                                                        0.0 
> %     6    0.6   0.5   0.4   0.6   0.1
> 2.  
> ip65 
> -44 
> -114 
> -97 
> .z114 
> -44 
> -65 
> .customer 
> .algx.net                                                          
> 0.0%     6    1.3   1.3   1.2   1.4   0.1
> 3. ???
>
>
> Trace from Savvis to XO IP space (65.44.114.97)
>
> 1.  
> scrubbed 
>                                                                                        0.0 
> %    38    0.4   0.4   0.3   0.5   0.1
> 2.  
> 64.41.199.129 
>                                                                                        0.0 
> %    37    1.0  24.0   0.6 330.2  80.4
> 3. hr1- 
> ge 
> -7 
> -47 
> .santaclarasc5 
> .savvis 
> .net                                                                 
> 0.0%    37    0.7   1.4   0.6  27.3   4.4
> 4. er1- 
> te 
> -1 
> -0 
> -0 
> .sanjose3equinix 
> .savvis 
> .net                                                              
> 0.0%    37    0.7   5.2   0.6 140.3  23.2
> 5. cr1- 
> tenge 
> -0 
> -7 
> -5 
> -0 
> .sanfrancisco 
> .savvis 
> .net                                                            
> 2.7%    37    2.9   4.0   2.6  16.6   2.5
> 6. cr2- 
> pos 
> -0 
> -0 
> -3 
> -3 
> .dallas 
> .savvis 
> .net 
>                                                                     
> 0.0%    37   42.6  43.1  42.3  51.4   1.4
> 7. dpr1- 
> ge 
> -4 
> -0 
> -0 
> .dallasequinix 
> .savvis 
> .net                                                               
> 0.0%    37   43.1  44.8  42.9  76.9   6.7
> 8. er1- 
> te 
> -2 
> -1 
> .dallasequinix 
> .savvis 
> .net                                                                  
> 0.0%    37   43.3  49.2  42.8 233.6  31.6
> 9.  
> 208.175.175.90 
>                                                                                       0.0 
> %    37   43.0  42.8  42.6  43.6   0.2
> 10.  
> 65.106.1.102 
>                                                                                        75.0 
> %    37   43.5  46.5  43.4  62.9   6.3
> 11.  
> 65.106.1.101 
>                                                                                         0.0 
> %    37   43.4  47.8  43.2 112.3  12.5
> 12.  
> 65.106.0.41 
>                                                                                          0.0 
> %    37   57.5  65.1  57.1 177.3  21.0
> 13.  
> 65.106.1.73 
>                                                                                          0.0 
> %    37   57.4  66.5  57.1 162.1  24.2
> 14. ???
>
> Trying to call into XO and they aren't even taking calls, they  
> mention something about network issues in Spokane. Any ideas as to  
> what is going on/ETA to fix?
>
> --Justin
>
>

-- 
Paul Norton
Systems Administrator
Neoverve - www.neoverve.com
Neoverve Blog - http://blog.neoverve.com/

_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages