[outages] Chronic packet loss/latency in LAX (4 providers potentially involved)

[Mark Keymer]

I don't think the issue is limited to just the providers listed.

Over the last several days we have been getting reports of long load 
times of sites hosted with us in Washington State. One specific customer 
is Just north of SF and is a comcast customer. Here is a traceroute from 
here before we worked with one of our upsteams today to route around 
cogent and currently the traffic is going over level 3 and it is better. 
I also had taken down that provider Last night for a while and Comcast 
took the traffic up to Seattle to my other provider were it was handed 
off. That link also worked very well. However Comcast to Cogent in 
Southern CA seems to be having issues best I can tell.

Sorry I don't think I have a MTR feed. I don't think any of my techs 
saved one yet going from us back down to my customers location.

Also i took out the beginning and ending of this for privacy. But the 
bulk of the traceroute is here. I find the hope from 68.86.87.6 to 
154.54.11.109 interesting in that is seems like there seems to be a good 
jump in latency there.

3 te-7-3-ur01.novato.ca.sfba.comcast.net (68.87.196.201) 26.191 ms
12.109 ms 10.970 ms
4 te-8-2-ur02.sanrafael.ca.sfba.comcast.net (68.87.192.145) 10.665 ms
14.917 ms 10.483 ms
5 te-9-3-ur01.sanrafael.ca.sfba.comcast.net (68.87.192.141) 24.769 ms
10.678 ms 10.807 ms
6 te-1-10-0-13-ar01.sfsutro.ca.sfba.comcast.net (68.87.226.122) 16.090
ms 18.981 ms 16.005 ms
7 he-1-7-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.153) 24.986 ms
16.506 ms 48.016 ms
8 pos-0-2-0-0-pe01.529bryant.ca.ibone.comcast.net (68.86.87.6) 14.379 ms
14.265 ms 13.569 ms
9 te0-7-0-12.ccr21.sjc04.atlas.cogentco.com (154.54.11.109) 59.591 ms
63.475 ms 63.213 ms
10 be2018.mpd22.sfo01.atlas.cogentco.com (154.54.28.81) 59.691 ms
be2015.ccr21.sfo01.atlas.cogentco.com (154.54.7.173) 65.196 ms
be2018.mpd22.sfo01.atlas.cogentco.com (154.54.28.81) 66.781 ms
11 te0-7-0-6.ccr22.sea01.atlas.cogentco.com (154.54.86.41) 74.862 ms
te0-7-0-0.ccr22.sea01.atlas.cogentco.com (154.54.86.37) 74.722 ms
te0-7-0-6.ccr22.sea01.atlas.cogentco.com (154.54.86.41) 74.747 ms
12 te0-6-0-4.ccr21.sea02.atlas.cogentco.com (154.54.85.182) 75.177 ms
te0-1-0-4.ccr21.sea02.atlas.cogentco.com (154.54.85.186) 75.314 ms
te0-1-0-5.ccr21.sea02.atlas.cogentco.com (154.54.85.178) 76.532 ms
13 te7-8.mag01.sea02.atlas.cogentco.com (154.54.41.213) 146.612 ms
te4-8.mag01.sea02.atlas.cogentco.com (154.54.41.209) 116.713 ms
te7-8.mag01.sea02.atlas.cogentco.com (154.54.41.213) 146.250 ms
14 38.104.127.10 (38.104.127.10) 74.407 ms 73.262 ms 75.154 ms
15 core01-grassi-eth1-6.net.pocketinet.com (64.185.97.125) 80.996 ms
80.480 ms 80.905 ms

I hope this info is helpful. I have more data I can provide. Been trying 
to work up some evidence on what is happening. And to try to pin point 
the problem that is causing issues. It seems to be intermittent. But 
like you said I think it does tend to be worse "Prime Time"

Sincerely,

Mark