[outages] Chronic packet loss/latency in LAX (4 providers potentially involved)

[Jesse]

Oh yes, I've been seeing this for some time, probably since mid-April.

At times we've seen Comcast prepending their AS multiple times on 
announcements through Level3, causing outbound to Comcast traffic 
via congested routes with Tata, rather than what would otherwise be the 
shorter, uncongested route via Level3. (Return traffic from Comcast in my 
case returns via Level3.)

For traffic from a server in San Francisco connected via Level3 and XO, 
connecting to two Comcast cable modems in San Francisco, I see traffic go 
out via XO, Tata, Comcast via LA, with what looks like major congestion, 
then back up to SF.  With the appearance of major congestion on the 
Tata<->Comcast link.

I don't have any traffic which visiblity traverses NTT or nLayer, so I 
couldn't say if the scope goes beyond the Tata-Comcast link.

It's particularly annoying receiving LA latency + 5-6% packet loss going 
to/from well connected locations a couple miles apart in SF. I'm 
experiencing it now composing this message over ssh.

Server in SF to Comcast in SF:
...
  5. te1-0-0d0.mcr1.fremont-ca.xo.net    0.0%   1.4   5.9   1.2  82.0  15.8
  6. vb1500.rar3.sanjose-ca.us.xo.net    0.0%  11.1   8.4   1.5  13.3   3.5
  7. 207.88.14.226.ptr.us.xo.net         6.0%  25.3   8.0   1.8  41.9  12.9
  8. if-10-12.icore1.SQN-SanJose.as6453 44.9%   6.0   7.0   2.0  14.9   4.3
  9. Vlan3260.icore2.SQN-SanJose.as6453  0.0%   3.1   6.6   2.3  13.5   3.4
10. if-4-2145.tcore1.SQN-SanJose.as645  0.0%   3.4   3.7   2.7   8.0   0.9
11. if-3-2.tcore2.LVW-LosAngeles.as645  0.0%  32.4  27.9  14.7  57.1   9.7
12. xe-0-2-0-0-pe01.onewilshire.ca.ibo  8.2%  39.4  30.3  28.3  72.3   7.1
13. te-2-0-0-6-cr01.losangeles.ca.ibon  6.1%  32.2  30.7  28.6  32.6   1.2
14. he-0-11-0-0-cr01.sacramento.ca.ibo  4.1%  30.6  31.5  29.6  33.3   1.0
15. he-0-4-0-0-ar01.oakland.ca.sfba.co 10.2%  30.5  32.3  30.4  34.3   1.1
16. te-8-4-ur04.santaclara.ca.sfba.com  8.2%  32.6  33.0  32.5  41.9   1.4
17. te-17-10-cdn08.sf19th.ca.sfba.comc  8.2%  43.0  60.5  31.2 830.2 117.4
...

The ComcastSF -> ServerInSF goes out via Level3 and sees no loss until 
the final hop, suggesting the problem is with the reverse path, above. 
(Most of the time, sometimes the first Comcast hops going out are very 
congested too, though that's not usually the case).

And here's two smokepings of the issue, in which you can clearly see the 
timing of the issue you mentioned. SF server to SF cable modem, 2mi apart.

http://cl.ly/image/040U3O2F0810 (last 30 hours)
http://cl.ly/image/1D053a0T1k2I (last 10 days)