[outages] peering in LA between XO and Savvis

[Justin Sharp]

After working with both vendors, it seems that a DDoS attack, aimed else 
where, which transited XO's peering router in LA was the root cause of 
this issue.

Seems to be cleared up now.

Justin Sharp wrote:
> We are seeing large (20-30%) packet loss in the forward and reverse 
> path from XO to Savvis in the Los Angeles peering point. This was 
> happening yesterday for about 2 hours, and has picked back up this 
> morning. This has made some sites hosted in Savvis unusable for us 
> (like Netsuite).
>
> Appropriate traces (some hops stripped to protect the innocent):
>
> From Draper UT to Savvis SC4 datacenter
>
> HOST: amds-mon                    Loss%   Snt   Last   Avg  Best  Wrst 
> StDev
>  1. stipped  0.0%    50    0.5   1.0   0.4   3.5   0.9
>  2. stripped      0.0%    50   29.1  10.7   0.2  38.6  13.9
>  3. stripped  0.0%    50   35.1  12.7   1.1  40.8  14.5
>  4. stripped  0.0%    50   43.1  21.8   2.9  48.9  15.8
>  5. p6-0-0d0.mar2.saltlake-ut.us  0.0%    50   40.0  25.2   3.1 155.9  
> 25.8
>  6. p3-2-0d0.rar2.la-ca.us.xo.ne  0.0%    50   21.1  41.2  17.1 128.8  
> 24.1
>  7. 207.88.12.157.ptr.us.xo.net   4.0%    50   26.6  34.9  17.8  60.9  
> 15.7
>  8. 207.88.12.158.ptr.us.xo.net   0.0%    50   25.4  34.1  17.0  65.2  
> 16.4
>  9. bpr2-so-6-1-0.losangeles.sav 30.0%    50   72.3  81.3  54.1 161.9  
> 25.5
> 10. cr2-gig-0-7-5-2.losangeles.s 32.0%    50   86.1  80.3  57.0 107.8  
> 17.0
> 11. cr2-pos0-0-0-0.sanfrancisco. 26.0%    50   76.7  90.9  65.5 116.8  
> 16.3
> 12. er1-te-2-0-1.sanjose3equinix 44.0%    50   70.7  86.6  66.2 112.8  
> 16.6
> 13. hr1-te-2-0-0.santaclarasc5.s 38.0%    50   70.8  86.2  66.6 116.5  
> 16.8
> 14. 216.34.2.226                 34.0%    50   92.1 174.0  66.8 823.9 
> 203.2
> 15. stripped                36.0%    50  109.1  90.3  66.1 118.5  17.4
> 16. end-point  44.0%    50  107.1  89.7  66.8 115.6  18.9
>
>
>
>> From  SC4 to same Draper UT network (reverse path, hop 16 is 
>> firewalled):
>
> HOST: prod-monitor                Loss%   Snt   Last   Avg  Best  Wrst 
> StDev
>
>  1. stripped                  0.0%    50    0.4   0.4   0.3   0.5   0.1
>  2. stripped                 0.0%    50  318.4  52.4   0.6 683.9 150.8
>  3. hr1-ge-7-47.santaclarasc5.sa  0.0%    50    0.9   3.3   0.6 128.2  
> 18.0
>  4. hr1-te-1-0-1.santaclarasc4.s  0.0%    50    1.0   0.9   0.6   
> 9.4   1.2
>  5. hr1-te-2-0-0.santaclarasc9.s  0.0%    50    1.1   1.8   0.6  
> 28.6   4.1
>  6. pr1-ge-4-0-0.SanJoseEquinix.  0.0%    50    1.3   7.1   1.0 128.2  
> 22.7
>  7. pr2-so-0-0-0.PaloAltoPaix.sa  0.0%    50   10.5   5.7   2.2  81.6  
> 12.5
>  8. p2-2.IR1.PaloAlto-CA.us.xo.n  0.0%    50    2.7   2.3   2.0   
> 3.1   0.2
>  9. 65.106.1.66.ptr.us.xo.net     0.0%    50    3.7   3.8   3.1  
> 13.0   1.4
> 10. 65.106.1.65.ptr.us.xo.net     0.0%    50   15.7   6.1   3.0  
> 35.9   7.8
> 11. p6-0-0.RAR1.LA-CA.us.xo.net  30.0%    50   56.4  57.8  49.9  
> 91.1   8.7
> 12. p0-0-0d0.rar2.la-ca.us.xo.ne 34.0%    50   55.1  63.6  43.9 157.8  
> 25.7
> 13. p1-0-0d0.mar2.saltlake-ut.us 42.0%    50   68.8  74.0  59.1 151.4  
> 16.8
> 14. p15-0.chr1.saltlake-ut.us.xo 36.0%    50   72.8  68.2  60.6  
> 76.3   3.6
> 15. stripped    36.0%    50   73.7  72.7  62.8  81.7   3.7
> 16. ???                          100.0    50    0.0   0.0   0.0   
> 0.0   0.0
>
>
> --Justin
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages