[outages] Fwd: above.net / he.net routing problems in LAX?

Thu Apr 26 21:09:00 EDT 2012

Rob,

Thanks for getting back to us.  You are indeed correct, the problem does 
appear to be inside the Corporate Colocation network.  Their NOC has 
been insisting this was upstream and the routing issues we're seeing as 
a result incorrectly implicated peers along the way.  I apologize for 
wasting your time, and again thank you for troubleshooting and responses.

-Erik

On 04/26/2012 07:40 PM, Rob Mosher wrote:
> You appear to be a customer of Corporate Colocation. Their equipment is
> the next hop in the traceroute, and the apparent point of failure.
> Perhaps contacting your provider before erroneously posting to an
> outages list may be prudent, as the point of failure does not seem to
> lie within HE, Abovenet, or GT-T, all of which have been named in your
> email.
>
> Also, GT-T should have sourced their trace from a non Equinix IX IP
> address, and it would have gotten further. This obviously did not help
> their diagnostics.
>
> Here is an MTR to 205.134.255.113, from an internal HE IP address.
> Please contact your provider, Corporate Colocation, for further
> troubleshooting.
>
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. gige-g3-8.core1.fmt1.he.net 0.0% 100 0.5 3.5 0.4 12.5 3.9
> 2. 10gigabitethernet1-1.core1.pao1.he.net 0.0% 99 0.8 3.5 0.7 11.5 3.6
> 3. 10gigabitethernet1-1.core1.lax1.he.net 0.0% 99 9.1 11.3 9.0 21.6 3.3
> 4. corporate-colocation-inc.gigabitethernet3- 70.4% 99 9.3 21.5 9.1
> 256.3 49.2
> 5. ???
>
>
> --
> Rob Mosher
> Senior Network and Software Engineer
> Hurricane Electric / AS6939
>
>
> On 4/26/2012 7:31 PM, Erik S. wrote:
>> Hi Rob,
>>
>> Thanks for the quick response. Sorry for the confusion, the
>> information I was provided at first was a little cloudy. We also use
>> GT-T/Packet Exchange and they're pointing this at above.net.
>>
>> Here's the response we received from PKX/GT-T regarding the routing
>> anomaly we're experiencing:
>>
>> -------- Original Message --------
>> Subject: GTT TT#(49181), SVC ID: PEX/NATIVE LAYER 3/9014-1 RE: Routing
>> issue preventing our network from reaching *
>> Date: Thu, 26 Apr 2012 21:52:27 +0000
>> From: GTT NOC <NOC at gt-t.net>
>>
>> From LAX core device the routes goes to abovenet for both the ip
>> address and dies within the above net network.
>> We will investigate with abovenet and update you shortly.
>>
>> cr1.lax1#traceroute 108.171.183.75
>>
>> Type Ctrl-C to abort.
>>
>> -------------------------------------------------------------------
>> Tracing the route to 108.171.183.75, 30 hops max, 40 byte packets
>> -------------------------------------------------------------------
>> TTL Hostname Probe1 Probe2 Probe3
>> 1 mpr1.lax2.us.above.net (206.223.123.71) 012.000 ms 001.000 ms
>> 001.000 ms
>> 2 * * *
>> 3 * * *
>> 4 * * *
>> 5 *
>>
>> -- End forwarded message --
>>
>> If they have not yet opened a ticket with you, this leads me to
>> believe there's a problem elsewhere and we're being given the run-around.
>>
>> The particular trace above is to one of our off-site boxes at rackspace.
>>
>> This is an issue that seems to be impacting multiple networks and
>> regions, so I'm not entirely convinced there's not something more
>> going on.
>>
>> For example, besides the IP referenced above, our 205.134.0.0 subnets
>> are also failing. I pulled this trace from Above.net's looking glass:
>>
>> traceroute to 205.134.255.113 (205.134.255.113), 30 hops max, 40 byte
>> packets
>> 1 xe-0-1-0.mpr1.lax12.us.above.net (64.125.31.189) 22.612 ms 0.421 ms
>> 0.368 ms
>> 2 10gigabitethernet1-3.core1.lax1.he.net (206.223.123.37) 0.944 ms
>> 12.178 ms 0.956 ms
>> 3 * * *
>> 4 * * *
>> 5 * * *
>> 6 * * *
>> 7 * * *
>> 8 * * *
>> 9 * * *
>> 10 * * *
>> 11 * * *
>> 12 * * *
>> 13 * * *
>> 14 * * *
>> 15 * * *
>> 16 * * *
>> 17 * * *
>> 18 * * *
>> 19 * * *
>> 20 * * *
>> 21 * * *
>> 22 * * *
>> 23 * * *
>> 24 * * *
>> 25 * * *
>> 26 * * *
>> 27 * corporate-colocation-inc.gigabitethernet3-16.core1.lax1.he.net
>> (72.52.77.38) 1.135 ms !H *
>> 28 * * *
>> 29 * * *
>>
>> Tracing out from 205.134.255.113 anywhere yields this:
>>
>> root at wczverify1 [~]# traceroute above.net
>> traceroute to above.net (64.125.228.35), 30 hops max, 40 byte packets
>> 1 74.124.198.1 (74.124.198.1) 0.461 ms 0.465 ms 0.530 ms
>> 2 gige-g2-19.core1.lax2.he.net (64.62.194.161) 0.363 ms 0.433 ms 0.509 ms
>> 3 * * *
>> 4 * * *
>> 5 * * *
>> 6 * * *
>>
>>
>> If you could shed any additional light on this issue for us I'd be
>> most appreciative.
>>
>> Thanks.
>>
>> Best Regards,
>>
>> Erik Soroka
>> Tier III System Administrator
>> InMotion Hosting, Inc.
>> eriks at inmotionhosting.com
>> 888-321-4678 (ext 834)
>> 757-416-6575 (int'l)
>>
>>
>> On 04/26/2012 07:20 PM, Rob Mosher wrote:
>>> Hi Erik,
>>>
>>> Where did you get the information that we were blaming above.net for
>>> anything? We are not aware of any issue between above.net and he.net,
>>> nor have we seen any tickets opened regarding this. Can you please
>>> include the source and destination IP addresses of the traceroute quoted
>>> below, as well as a traceroute in the opposite direction if possible?
>>>
>>> As far as I can tell, everything looks clean here.
>>>
>>> root at tserv15:~# tcptraceroute lg.above.net
>>> traceroute to lg.above.net (64.124.253.67), 30 hops max, 60 byte packets
>>> 1 gige-g4-6.core1.lax1.he.net (66.220.18.41) 7.683 ms 7.725 ms
>>> 7.821 ms
>>> 2 mpr1.lax2.us.above.net (206.223.123.71) 0.781 ms 0.766 ms 0.767 ms
>>> 3 xe-3-0-0.cr1.lax112.us.above.net (64.125.30.17) 1.055 ms 1.057
>>> ms 1.293 ms
>>> 4 xe-2-1-0.cr1.sjc2.us.above.net (64.125.24.18) 10.125 ms 10.129
>>> ms 10.120 ms
>>> 5 xe-1-1-0.er1.sjc2.us.above.net (64.125.26.198) 9.815 ms 9.800 ms
>>> 9.800 ms
>>> 6 f1-1.itr3.sjc2.us.corp.above.net (64.124.248.122) 10.239 ms
>>> 10.002 ms 9.995 ms
>>> 7 lg.above.net (64.124.253.67) 10.227 ms 10.338 ms 10.267 ms
>>>
>>> --
>>> Rob Mosher
>>> Senior Network and Software Engineer
>>> Hurricane Electric / AS6939
>>>
>>>
>>>> ---------- Forwarded message ----------
>>>> From: "Erik S." <eriks at inmotionhosting.com
>>>> <mailto:eriks at inmotionhosting.com>>
>>>> Date: Apr 26, 2012 6:06 PM
>>>> Subject: [outages] above.net <http://above.net> / he.net
>>>> <http://he.net> routing problems in LAX?
>>>> To: "outages at outages.org <mailto:outages at outages.org>"
>>>> <outages at outages.org <mailto:outages at outages.org>>
>>>>
>>>> We're seeing major routing issues in Los Angeles with peering from
>>>> HE.net <-> Above.net, specifically after mpr1.lax2.us.above.net
>>>> <http://mpr1.lax2.us.above.net>.
>>>>
>>>> Above.net's looking glass in LA shows traces dying right after he.net
>>>> <http://he.net>,
>>>>
>>>> 1 xe-0-1-0.cr2.lax112.us.above.net
>>>> <http://xe-0-1-0.cr2.lax112.us.above.net> (64.125.30.250
>>>> <tel:%2864.125.30.250>) 0.780 ms 0.571 ms 0.479 ms
>>>> 2 xe-8-0-0.mpr1.lax12.us.above.net
>>>> <http://xe-8-0-0.mpr1.lax12.us.above.net> (64.125.30.30) 0.393 ms
>>>> 0.483 ms 0.404 ms
>>>> 3 10gigabitethernet1-3.core1.lax1.he.net
>>>> <http://10gigabitethernet1-3.core1.lax1.he.net> (206.223.123.37)
>>>> 0.884 ms 4.619 ms 1.100 ms
>>>> 4 * * *
>>>>
>>>> Both sides seem to be blaming the other. Is anyone routing via
>>>> above.net <http://above.net> or he.net <http://he.net> seeing similar
>>>> issues out west?
>>>>
>>>> Best Regards,
>>>>
>>>> Erik Soroka
>>>> Tier III System Administrator
>>>> InMotion Hosting, Inc.
>>>> eriks at inmotionhosting.com <mailto:eriks at inmotionhosting.com>
>>>>
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org <mailto:Outages at outages.org>
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>
>