[outages] comcast to Global Crossing issues
Blake Pfankuch
bpfankuch at cpgreeley.com
Wed Sep 21 00:58:13 EDT 2011
Correct. Getting to the other side has taken a bit of doing, however traceroutes coming back to my house don't come across GBLX, destination in 75.151.89.0/24. Anything going from local comcast network to this remote location crossing gblx seems to fail. From another server in denver over tinet, looks good (no gblx touching) and same from phoenix. I'm not sure where the issue lies, just appears to me to from comcast destined to gblx, hense asking if anyone else sees something weird.
2 62-233-127-181.as20860.net (62.233.127.181) 1.259 ms 1.260 ms 1.252 ms
3 202.core1.thn.as20860.net (62.233.127.11) 20.223 ms 20.466 ms 20.460 ms
4 xe-10-2-0.londra32.lon.seabone.net (89.221.43.105) 1.214 ms 1.208 ms 1.20 1 ms
5 te4-4.newyork51.new.seabone.net (195.22.216.211) 70.649 ms te4-3.newyork51.new.seabone.net (195.22.195.3) 72.886 ms te4-4.newyork51.new.seabone.net (195.2 2.216.211) 70.647 ms
6 be-10-204-pe01.111eighthave.ny.ibone.comcast.net (75.149.230.77) 71.120 ms 71.365 ms 71.364 ms
7 pos-1-9-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.86.41) 70.609 ms 70.6 03 ms 70.589 ms
8 pos-0-6-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.88.154) 91.822 ms 91.816 ms 91.803 ms
9 pos-1-7-0-0-cr01.chicago.il.ibone.comcast.net (68.86.88.45) 92.044 ms 91.6 53 ms 92.559 ms
10 * * *
11 * * *
You are correct something is always broken, however OOB is not something that's justifiable in this case :)
-----Original Message-----
From: Jeremy Chadwick [mailto:outages at jdc.parodius.com]
Sent: Tuesday, September 20, 2011 10:33 PM
To: Blake Pfankuch
Cc: outages at outages.org
Subject: Re: [outages] comcast to Global Crossing issues
Are you absolutely certain this is the fault of GBLX? Your traceroute below is only one direction -- to get a clear picture you need traceroutes from *both* directions; return packets may be taking an entirely different route and that could be what's broken (and would render as such in your traceroute). Most of the time on the Internet routing IS NOT symmetrical. You're a network engineer though so I assume you know this.
78.129.136.1 reachable from Comcast in the California Bay Area. This isn't shown in the below mtr because of what appears to be ACLs somewhere near/around the border of the network shown below blocking ICMP time exceeded. These egress packets obviously don't use GBLX; cannot check ingress, obviously. ICMP echo/echo reply to the endpoint does work.
Host Loss% Snt Rcv Last Avg Best Wrst
1. gw.home.lan 0.0% 14 14 0.3 0.4 0.2 0.6
2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 14 14 25.0 24.5 9.4 40.6
3. te-7-3-ur04.santaclara.ca.sfba.comcast.net 0.0% 13 13 9.7 15.3 8.9 44.8
4. te-1-10-0-7-ar01.oakland.ca.sfba.comcast.ne 0.0% 13 13 12.4 13.7 10.9 24.5
5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 13 13 13.3 15.2 13.0 30.3
6. pos-0-2-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 13 13 16.8 17.5 15.3 21.3
7. pos-0-4-0-0-pe01.529bryant.ca.ibone.comcast 0.0% 13 13 17.2 18.8 16.6 32.2
8. as6762-pe01.350ecermak.il.ibone.comcast.net 0.0% 13 13 16.7 17.5 15.8 21.7
9. xe-8-0-0.londra32.lon.seabone.net 0.0% 13 13 159.1 160.8 157.9 175.5
10. iomart.londra32.lon.seabone.net 0.0% 13 13 161.0 160.1 158.5 161.1
11. 62-233-127-174.as20860.net 0.0% 13 13 160.8 160.9 159.7 162.5
12. 87.117.211.33 0.0% 13 13 162.0 165.0 158.9 190.9
13. ???
$ ping 78.129.136.1
PING 78.129.136.1 (78.129.136.1): 56 data bytes
64 bytes from 78.129.136.1: icmp_seq=0 ttl=240 time=212.176 ms
64 bytes from 78.129.136.1: icmp_seq=1 ttl=240 time=200.820 ms
64 bytes from 78.129.136.1: icmp_seq=2 ttl=240 time=182.815 ms
64 bytes from 78.129.136.1: icmp_seq=3 ttl=240 time=171.383 ms
64 bytes from 78.129.136.1: icmp_seq=4 ttl=240 time=164.347 ms
64 bytes from 78.129.136.1: icmp_seq=5 ttl=240 time=165.554 ms
64 bytes from 78.129.136.1: icmp_seq=6 ttl=240 time=163.315 ms
64 bytes from 78.129.136.1: icmp_seq=7 ttl=240 time=163.348 ms
64 bytes from 78.129.136.1: icmp_seq=8 ttl=240 time=201.289 ms ^C
--- 78.129.136.1 ping statistics ---
9 packets transmitted, 9 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 163.315/180.561/212.176/18.290 ms
>From my co-location provider (in Santa Clara, CA) to said IP. Again, no use of GBLX on the egress path, but this time ACLs permit ICMP time exceeded.
Host Loss% Snt Rcv Last Avg Best Wrst
1. 72.20.98.65 0.0% 15 15 0.4 0.5 0.3 1.3
2. er0sc2.bayarea.net.bayarea.net 0.0% 15 15 0.3 0.3 0.2 0.5
3. ge-11-0-1.er2.sjc2.above.net 0.0% 15 15 0.5 0.5 0.4 0.6
4. xe-1-1-0.mpr4.sjc7.us.above.net 0.0% 14 14 14.6 6.3 1.0 53.7
5. xe-0-1-0-6.sjc10.ip4.tinet.net 0.0% 14 14 1.3 5.0 1.0 53.9
6. xe-11-0-0.lon25.ip4.tinet.net 0.0% 14 14 142.5 144.8 142.5 173.2
7. iomart-hosting-gw.ip4.tinet.net 0.0% 14 14 143.6 182.0 143.5 280.8
8. 78.129.136.1 0.0% 14 14 148.0 147.7 145.0 174.1
Reminder to readers: the Internet is broken 24x7x365, and I'm quite serious. For mission-critical services that need maintenance, OOB
(out-of-band) is required. If that means dialing internationally into a modem in London to do some device config changes then that's what it means. Rock the POTS lines, don't be afraid.
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
On Wed, Sep 21, 2011 at 03:58:02AM +0000, Blake Pfankuch wrote:
> Anyone else seeing any issues going Comcast to Global Crossing this evening? I was smack in the middle of some work on an EU server when it disconnected my ssh session. Thought I broke it, however I cannot load the datacenter website from home. From the office everything shows fine. From home traceroute shows fail past 208.178.62.141 going to 78.129.136.0/24 or 79.129.226.0/24, and I cannot load http://ipstats.globalcrossing.net either. Down for me says it's just me, but I cannot get to any of the affected services from one of my customer networks on Comcast either.
>
> 4 11 ms 9 ms 11 ms te-8-4-ur02.greeley.co.denver.comcast.net [68.86.128.22]
> 5 14 ms 11 ms 16 ms te-0-8-0-5-ar02.aurora.co.denver.comcast.net [68.86.103.82]
> 6 11 ms 16 ms 16 ms pos-3-7-0-0-cr01.denver.co.ibone.comcast.net [68.86.91.101]
> 7 33 ms 39 ms 29 ms pos-2-10-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.87.186]
> 8 36 ms 27 ms 31 ms 208.178.62.141
> 9 * * * Request timed out.
> 10 * * * Request timed out.
>
>
> Blake Pfankuch
> bpfankuch at cpgreeley.com<mailto:bpfankuch at cpgreeley.com>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
More information about the Outages
mailing list