[c-nsp] transatlantic Internet latency

Richard A Steenbergen ras at e-gerbil.net
Fri Jul 29 19:13:20 EDT 2005


On Fri, Jul 29, 2005 at 05:00:31PM -0400, Adam Greene wrote:
> Hi,
> 
> We have a customer who's experiencing high latency on IP traffic between New
> York (where we are) and their server in New Zealand. As far as I can tell,
> this is normal behavior.

You know traceroute is probably the second most dangerous tool on the 
Internet in the hands of someone who doesn't know what to do with it, 
right after those $#%^&*ing personal firewall programs that help people 
detect and complain about their web browsing traffic and DNS requests.

> As far as I can tell, latency jumps on the transition from UUNet to Global
> Crossing, then again at the leap from Global Crossing to Asia Netcom. Can
> anyone tell from experience if this is normal behavior? I assume the GBLX /
> asianetcom transition is transatlantic fiber ...

Trans-pacific.

So the bottom line is that you're confused by the lack of DNS on hop 11 
and the transition to a higher latency on hop 12? Lets review:

> Here's a snip from the traceroute:
> 
>  7   37ms     1/ 100 =  1%     1/ 100 =  1%  500.POS4-2.GW1.NYC9.ALTER.NET
> [208.192.182.125]

This is your providers' customer edge router. This is what is known as a 
customer demarc, usually a /30 on a point to point circuit (which this is, 
a PoS interface), and the other side of this /30 is your router:

Name:    hvdata-gw.customer.alter.net
Address:  208.192.182.126

Note that you start out coming in to this with 37ms of rtt latency, which 
either means that the other end of this SONET circuit is really far away, 
that the circuit is congested, or that someone is running out of CPU to 
process TTL expired responses.

>                                 0/ 100 =  0%   |
>   8   37ms     2/ 100 =  2%     2/ 100 =  2%  0.so-3-0-0.XL1.NYC9.ALTER.NET
> [152.63.99.178]
>                                 0/ 100 =  0%   |
>   9   38ms     0/ 100 =  0%     0/ 100 =  0%  0.so-7-0-0.XL1.NYC8.ALTER.NET
> [152.63.0.37]

Even without knowing anything about UU's architecture, you can tell that 
you are still in the same region, just a different building or POP, and 
since the latency is staying consistant you know that it isn't a router 
CPU exhaustion problem. Either you have a congested local pipe, or the 
other end of this circuit runs somewhere far away.

>                                 0/ 100 =  0%   |
>  10   38ms     0/ 100 =  0%     0/ 100 =  0%  POS6-0.BR3.NYC8.ALTER.NET
> [152.63.19.53]

Now we're at a border router. One of the nice things about networks at UU 
is that they are very structured, you can pretty much be guarantee that 
customers won't be turning up on BR's and peers won't be turning up on 
GW's etc. That means the next hop is probably going to be another network.

>                                 0/ 100 =  0%   |
>  11   41ms     0/ 100 =  0%     0/ 100 =  0%  204.255.168.62
>                                 0/ 100 =  0%   |

Ah and it is, but you have no way to know that. It seems that either 
through policy or laziness someone has neglected to put reverse DNS on 
the remote side of this /30. Of course you can always go looking at the 
other side for more info:

Name:    POS1-1.BR3.NYC8.ALTER.NET
Address:  204.255.168.61

You can also see that the /30 came out of UUNet's IP space, 204.252.0.0/14 
to be precise. Not immediately helpful in figuring out who the other side 
is, but it is 100% clear that this is a peer.

>  12  113ms     3/ 100 =  3%     3/ 100 =  3%
> so1-1-0-2488M.ar1.SJC2.gblx.net [67.17.64.65]

Ok well now you know who the other side of that last /30 was, Global 
Crossing. The reason that you don't see any routing infrastructure inside 
GBLX's network is because it is hidden within an MPLS LSP that is set to 
not decrement the TTL as the label is switched through the network.

However, you can now tell that you are in SJC. A quick word about parsing 
location identifiers. The most common types out there are:

1) Airport codes (like SJC)
2) CLLI geographic+geopolitical codes (like sjnsca)
3) Pseudo-airport codes (like NYC instead of LGA or JFK)
4) Random garbage from deranged minds

Luckily enough for you, GX and UU both fall into the category of "mostly" 
real airport codes, with some fake ones thrown in for good measure just to 
annoy people who don't know what to look for.

SJC = San Jose California, a real airport code
NYC = New York NY, a fake airport code, but one that is easy enough to 
figure out.

This is crossing North America. By subjecting the previous latency of 41ms 
from 113ms you can see that this hop has added 72ms of extra latency. 
Perfectly normal for going that distance.

>                                 0/ 100 =  0%   |
>  13  110ms     2/ 100 =  2%     2/ 100 =  2%
> ANC-San-Jose-3.ge-2-3-1.ar1.SJC2.gblx.net [64.215.184.246]

Hey look they confirmed it for you. Since you know that UU-GX is a peering 
relationship, and you are UU's customer, you can pretty much infer that 
ANC is a customer. This specific hop that you're looking at is another 
customer demarc, between GX and ANC. You can see that the other side of 
the /30 is:

Name:    ge-2-3-1.ar1.SJC2.gblx.net
Address:  64.215.184.245

>                                 0/ 100 =  0%   |
>  14  262ms     0/ 100 =  0%     0/ 100 =  0%  po3-0.gw1.akl1.asianetcom.net
> [202.147.55.213]

So, now you know that ANC is Asia NetCom. You also know that the last 
router in San Jose was theirs, so you now know that you have traveled 
262ms - 110ms = 152ms on their network.

>                                 0/ 100 =  0%   |
>  15  237ms     2/ 100 =  2%     2/ 100 =  2%  61.14.140.83
>                                 0/ 100 =  0%   |
>  16  265ms     0/ 100 =  0%     0/ 100 =  0%  vl5.cr1.akl.tranzpeer.net
> [202.180.64.118]

Yada yada yada.

Note that this entire process misses one of the most important elements of 
deciphering a route, the reverse path. Of course since I don't have that 
data, and it appears to be irrelevent anyways, I'm just going to ignore 
that. Answer is, perfectly normal latency.

> Just trying to find something to tell the customer short of hosting the
> server here in NY.

The fiber paths COULD be a tiny bit better. 72ms is actually on the high 
side for this route, a better route could come out somewhere in the 66ms 
range. What you're actually seeing is 76ms on the GX NYC->SJC rtt and 
something lower on the UU path of SJC->NYC, which balances out to 72ms 
when you consider both sides of the path. If you're a GX customer you 
probably understand why that is, but that is a discussion for another day. 
:)

The best you could hope for even with perfectly optimized fiber paths is 
only a few ms of improvement. If you want to make it significantly better, 
either move one of the endpoints, or make light move through fiber faster. 
:)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)



More information about the cisco-nsp mailing list