[outages] peering in LA between XO and Savvis

[Justin Sharp]

We are seeing large (20-30%) packet loss in the forward and reverse path 
from XO to Savvis in the Los Angeles peering point. This was happening 
yesterday for about 2 hours, and has picked back up this morning. This 
has made some sites hosted in Savvis unusable for us (like Netsuite).

Appropriate traces (some hops stripped to protect the innocent):

 From Draper UT to Savvis SC4 datacenter

HOST: amds-mon                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. stipped  0.0%    50    0.5   1.0   0.4   3.5   0.9
  2. stripped      0.0%    50   29.1  10.7   0.2  38.6  13.9
  3. stripped  0.0%    50   35.1  12.7   1.1  40.8  14.5
  4. stripped  0.0%    50   43.1  21.8   2.9  48.9  15.8
  5. p6-0-0d0.mar2.saltlake-ut.us  0.0%    50   40.0  25.2   3.1 155.9  25.8
  6. p3-2-0d0.rar2.la-ca.us.xo.ne  0.0%    50   21.1  41.2  17.1 128.8  24.1
  7. 207.88.12.157.ptr.us.xo.net   4.0%    50   26.6  34.9  17.8  60.9  15.7
  8. 207.88.12.158.ptr.us.xo.net   0.0%    50   25.4  34.1  17.0  65.2  16.4
  9. bpr2-so-6-1-0.losangeles.sav 30.0%    50   72.3  81.3  54.1 161.9  25.5
 10. cr2-gig-0-7-5-2.losangeles.s 32.0%    50   86.1  80.3  57.0 107.8  17.0
 11. cr2-pos0-0-0-0.sanfrancisco. 26.0%    50   76.7  90.9  65.5 116.8  16.3
 12. er1-te-2-0-1.sanjose3equinix 44.0%    50   70.7  86.6  66.2 112.8  16.6
 13. hr1-te-2-0-0.santaclarasc5.s 38.0%    50   70.8  86.2  66.6 116.5  16.8
 14. 216.34.2.226                 34.0%    50   92.1 174.0  66.8 823.9 203.2
 15. stripped                36.0%    50  109.1  90.3  66.1 118.5  17.4
 16. end-point  44.0%    50  107.1  89.7  66.8 115.6  18.9

 

>From  SC4 to same Draper UT network (reverse path, hop 16 is firewalled):

HOST: prod-monitor                Loss%   Snt   Last   Avg  Best  Wrst StDev

  1. stripped                  0.0%    50    0.4   0.4   0.3   0.5   0.1
  2. stripped                 0.0%    50  318.4  52.4   0.6 683.9 150.8
  3. hr1-ge-7-47.santaclarasc5.sa  0.0%    50    0.9   3.3   0.6 128.2  18.0
  4. hr1-te-1-0-1.santaclarasc4.s  0.0%    50    1.0   0.9   0.6   9.4   1.2
  5. hr1-te-2-0-0.santaclarasc9.s  0.0%    50    1.1   1.8   0.6  28.6   4.1
  6. pr1-ge-4-0-0.SanJoseEquinix.  0.0%    50    1.3   7.1   1.0 128.2  22.7
  7. pr2-so-0-0-0.PaloAltoPaix.sa  0.0%    50   10.5   5.7   2.2  81.6  12.5
  8. p2-2.IR1.PaloAlto-CA.us.xo.n  0.0%    50    2.7   2.3   2.0   3.1   0.2
  9. 65.106.1.66.ptr.us.xo.net     0.0%    50    3.7   3.8   3.1  13.0   1.4
 10. 65.106.1.65.ptr.us.xo.net     0.0%    50   15.7   6.1   3.0  35.9   7.8
 11. p6-0-0.RAR1.LA-CA.us.xo.net  30.0%    50   56.4  57.8  49.9  91.1   8.7
 12. p0-0-0d0.rar2.la-ca.us.xo.ne 34.0%    50   55.1  63.6  43.9 157.8  25.7
 13. p1-0-0d0.mar2.saltlake-ut.us 42.0%    50   68.8  74.0  59.1 151.4  16.8
 14. p15-0.chr1.saltlake-ut.us.xo 36.0%    50   72.8  68.2  60.6  76.3   3.6
 15. stripped    36.0%    50   73.7  72.7  62.8  81.7   3.7
 16. ???                          100.0    50    0.0   0.0   0.0   0.0   0.0


--Justin