[outages] SF Bay: Comcast/TATA issue or GBLX/nLayer issue?
Jeremy Chadwick
outages at jdc.parodius.com
Thu Nov 3 18:36:38 EDT 2011
Not sure when this started, could have been going on for hours. I just
got in to work and attempted to RDP into my home machine -- no go.
The asymmetric nature of routing makes this a little difficult to
determine where the actual loss is, but I imagine what's shown in WinMTR
is correct -- ingress packets are (were, keep reading) getting lost at
some point. Note in the ingress mtr the Worst field. Also note that on
the ingress side, packets were going from San Jose->Atlanta->Fort
Worth->Los Angeles.
I can't tell if this was a Comcast<->TATA Communications problem, or if
it was a GBLX<->nLayer issue. I'm inclined to believe the latter.
And as usual these days, the instant I begin writing about the problem,
it stops. Whatever this was, however, was pretty major. I wish I had a
start time. End time was approximately 15:28 PDT. I have not checked
BGPlay to look for routing/reconvergence.
Egress (work -> home):
|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| {snip} |
| xe1-8.core2.sv1.layer42.net - 0 | 254 | 254 | 3 | 4 | 49 | 3 |
| sjo-bb1-link.telia.net - 0 | 254 | 254 | 2 | 4 | 30 | 3 |
| tata-ic-140123-sjo-bb1.c.telia.net - 31 | 115 | 80 | 3 | 7 | 16 | 3 |
|if-4-3254.tcore2.PDI-PaloAlto.as6453.net - 1 | 250 | 249 | 3 | 7 | 91 | 4 |
| if-2-2.tcore1.PDI-PaloAlto.as6453.net - 0 | 254 | 254 | 3 | 5 | 63 | 5 |
| 66.198.127.14 - 34 | 106 | 70 | 160 | 163 | 167 | 164 |
|pos-1-10-0-0-cr01.sanjose.ca.ibone.comcast.net - 35 | 108 | 71 | 149 | 160 | 165 | 149 |
|pos-1-8-0-0-cr01.sacramento.ca.ibone.comcast.net - 25 | 130 | 98 | 152 | 167 | 171 | 152 |
|pos-0-13-0-0-ar01.oakland.ca.sfba.comcast.net - 33 | 112 | 76 | 0 | 168 | 171 | 162 |
|te-9-8-ur04.santaclara.ca.sfba.comcast.net - 39 | 99 | 61 | 0 | 166 | 168 | 166 |
| 68.85.191.250 - 33 | 111 | 75 | 167 | 188 | 295 | 167 |
| c-67-180-84-87.hsd1.ca.comcast.net - 27 | 124 | 91 | 0 | 180 | 203 | 174 |
|________________________________________________|______|______|______|______|______|______|
Ingress (home -> work):
Packets Pings
Host Loss% Snt Rcv Last Avg Best Wrst
1. gw.home.lan 0.0% 72 72 0.3 0.6 0.2 1.1
2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 71 71 24.7 55.5 12.1 1484.
3. te-5-5-ur04.santaclara.ca.sfba.comcast.net 0.0% 71 71 9.0 12.6 8.4 49.8
4. te-0-4-0-2-ar01.oakland.ca.sfba.comcast.net 0.0% 71 71 10.8 14.6 10.5 47.7
5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 71 71 13.2 15.3 11.9 35.6
6. pos-0-4-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 71 71 16.6 18.4 14.7 59.7
7. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comca 0.0% 71 71 22.1 24.0 19.5 52.6
8. 208.178.58.1 0.0% 71 71 19.8 24.6 17.7 58.4
9. ae9-20G.scr3.SNV2.gblx.net 0.0% 71 71 20.5 22.8 18.8 53.3
10. ae0.ar4.ATL2.gblx.net 0.0% 71 71 77.5 76.9 73.8 101.3
11. xe-3-0-4.ar1.atl1.us.nlayer.net 2.8% 71 69 131.9 150.7 76.2 253.8
12. ae0-70g.cr1.atl1.us.nlayer.net 0.0% 71 71 75.1 113.2 73.7 172.0
13. xe-2-2-1.cr1.dfw1.us.nlayer.net 0.0% 71 71 89.9 115.7 73.8 192.8
14. xe-5-3-0.cr1.lax1.us.nlayer.net 0.0% 71 71 80.6 117.2 74.3 195.8
15. xe-1-2-0.cr1.sjc1.us.nlayer.net 0.0% 71 71 78.8 121.3 76.5 209.1
16. ae1-50g.cr1.pao1.us.nlayer.net 0.0% 71 71 77.7 118.1 77.1 197.1
17. as8121.ae0-3001.cr1.pao1.us.nlayer.net 1.4% 71 70 77.9 118.6 77.1 182.2
18. xe1-4.core1.svk.layer42.net 0.0% 71 71 78.7 117.0 76.5 174.9
19. ???
Below is how things look like now:
Egress (work -> home): looks the same as the above WinMTR, sans packet loss.
Ingress (home -> work):
Packets Pings
Host Loss% Snt Rcv Last Avg Best Wrst
1. gw.home.lan 0.0% 6 6 0.2 0.4 0.2 0.9
2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 5 5 12.4 21.7 12.4 27.4
3. te-5-5-ur04.santaclara.ca.sfba.comcast.net 0.0% 5 5 10.5 10.2 9.7 10.6
4. te-0-4-0-2-ar01.oakland.ca.sfba.comcast.net 0.0% 5 5 12.1 11.6 11.0 12.1
5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 5 5 14.5 14.7 12.9 15.8
6. pos-0-4-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 5 5 14.7 16.4 14.7 17.6
7. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comca 0.0% 5 5 20.9 22.2 20.8 24.7
8. 208.178.58.1 0.0% 5 5 25.3 26.7 18.8 41.6
9. ae9-20G.scr3.SNV2.gblx.net 0.0% 5 5 18.7 19.8 18.7 20.7
10. te6-4-10G.ar1.PAO2.gblx.net 0.0% 5 5 19.5 20.5 19.5 21.7
11. xe-2-0-4.ar1.pao1.us.nlayer.net 0.0% 5 5 23.7 23.4 21.1 27.1
12. ae0-80g.cr1.pao1.us.nlayer.net 0.0% 5 5 18.7 20.1 18.7 21.1
13. as8121.ae0-3001.cr1.pao1.us.nlayer.net 0.0% 5 5 22.8 23.2 20.0 26.6
14. xe1-4.core1.svk.layer42.net 0.0% 5 5 21.1 22.9 19.3 34.9
15. ???
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
More information about the Outages
mailing list