[outages] comcast to Global Crossing issues
Jeremy Chadwick
outages at jdc.parodius.com
Wed Sep 21 00:33:14 EDT 2011
Are you absolutely certain this is the fault of GBLX? Your traceroute
below is only one direction -- to get a clear picture you need
traceroutes from *both* directions; return packets may be taking an
entirely different route and that could be what's broken (and would
render as such in your traceroute). Most of the time on the Internet
routing IS NOT symmetrical. You're a network engineer though so I
assume you know this.
78.129.136.1 reachable from Comcast in the California Bay Area. This
isn't shown in the below mtr because of what appears to be ACLs
somewhere near/around the border of the network shown below blocking
ICMP time exceeded. These egress packets obviously don't use GBLX;
cannot check ingress, obviously. ICMP echo/echo reply to the endpoint
does work.
Host Loss% Snt Rcv Last Avg Best Wrst
1. gw.home.lan 0.0% 14 14 0.3 0.4 0.2 0.6
2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 14 14 25.0 24.5 9.4 40.6
3. te-7-3-ur04.santaclara.ca.sfba.comcast.net 0.0% 13 13 9.7 15.3 8.9 44.8
4. te-1-10-0-7-ar01.oakland.ca.sfba.comcast.ne 0.0% 13 13 12.4 13.7 10.9 24.5
5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 13 13 13.3 15.2 13.0 30.3
6. pos-0-2-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 13 13 16.8 17.5 15.3 21.3
7. pos-0-4-0-0-pe01.529bryant.ca.ibone.comcast 0.0% 13 13 17.2 18.8 16.6 32.2
8. as6762-pe01.350ecermak.il.ibone.comcast.net 0.0% 13 13 16.7 17.5 15.8 21.7
9. xe-8-0-0.londra32.lon.seabone.net 0.0% 13 13 159.1 160.8 157.9 175.5
10. iomart.londra32.lon.seabone.net 0.0% 13 13 161.0 160.1 158.5 161.1
11. 62-233-127-174.as20860.net 0.0% 13 13 160.8 160.9 159.7 162.5
12. 87.117.211.33 0.0% 13 13 162.0 165.0 158.9 190.9
13. ???
$ ping 78.129.136.1
PING 78.129.136.1 (78.129.136.1): 56 data bytes
64 bytes from 78.129.136.1: icmp_seq=0 ttl=240 time=212.176 ms
64 bytes from 78.129.136.1: icmp_seq=1 ttl=240 time=200.820 ms
64 bytes from 78.129.136.1: icmp_seq=2 ttl=240 time=182.815 ms
64 bytes from 78.129.136.1: icmp_seq=3 ttl=240 time=171.383 ms
64 bytes from 78.129.136.1: icmp_seq=4 ttl=240 time=164.347 ms
64 bytes from 78.129.136.1: icmp_seq=5 ttl=240 time=165.554 ms
64 bytes from 78.129.136.1: icmp_seq=6 ttl=240 time=163.315 ms
64 bytes from 78.129.136.1: icmp_seq=7 ttl=240 time=163.348 ms
64 bytes from 78.129.136.1: icmp_seq=8 ttl=240 time=201.289 ms
^C
--- 78.129.136.1 ping statistics ---
9 packets transmitted, 9 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 163.315/180.561/212.176/18.290 ms
>From my co-location provider (in Santa Clara, CA) to said IP. Again, no
use of GBLX on the egress path, but this time ACLs permit ICMP time
exceeded.
Host Loss% Snt Rcv Last Avg Best Wrst
1. 72.20.98.65 0.0% 15 15 0.4 0.5 0.3 1.3
2. er0sc2.bayarea.net.bayarea.net 0.0% 15 15 0.3 0.3 0.2 0.5
3. ge-11-0-1.er2.sjc2.above.net 0.0% 15 15 0.5 0.5 0.4 0.6
4. xe-1-1-0.mpr4.sjc7.us.above.net 0.0% 14 14 14.6 6.3 1.0 53.7
5. xe-0-1-0-6.sjc10.ip4.tinet.net 0.0% 14 14 1.3 5.0 1.0 53.9
6. xe-11-0-0.lon25.ip4.tinet.net 0.0% 14 14 142.5 144.8 142.5 173.2
7. iomart-hosting-gw.ip4.tinet.net 0.0% 14 14 143.6 182.0 143.5 280.8
8. 78.129.136.1 0.0% 14 14 148.0 147.7 145.0 174.1
Reminder to readers: the Internet is broken 24x7x365, and I'm quite
serious. For mission-critical services that need maintenance, OOB
(out-of-band) is required. If that means dialing internationally into a
modem in London to do some device config changes then that's what it
means. Rock the POTS lines, don't be afraid.
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
On Wed, Sep 21, 2011 at 03:58:02AM +0000, Blake Pfankuch wrote:
> Anyone else seeing any issues going Comcast to Global Crossing this evening? I was smack in the middle of some work on an EU server when it disconnected my ssh session. Thought I broke it, however I cannot load the datacenter website from home. From the office everything shows fine. From home traceroute shows fail past 208.178.62.141 going to 78.129.136.0/24 or 79.129.226.0/24, and I cannot load http://ipstats.globalcrossing.net either. Down for me says it's just me, but I cannot get to any of the affected services from one of my customer networks on Comcast either.
>
> 4 11 ms 9 ms 11 ms te-8-4-ur02.greeley.co.denver.comcast.net [68.86.128.22]
> 5 14 ms 11 ms 16 ms te-0-8-0-5-ar02.aurora.co.denver.comcast.net [68.86.103.82]
> 6 11 ms 16 ms 16 ms pos-3-7-0-0-cr01.denver.co.ibone.comcast.net [68.86.91.101]
> 7 33 ms 39 ms 29 ms pos-2-10-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.87.186]
> 8 36 ms 27 ms 31 ms 208.178.62.141
> 9 * * * Request timed out.
> 10 * * * Request timed out.
>
>
> Blake Pfankuch
> bpfankuch at cpgreeley.com<mailto:bpfankuch at cpgreeley.com>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
More information about the Outages
mailing list