[c-nsp] transatlantic Internet latency
Richard A Steenbergen
ras at e-gerbil.net
Fri Jul 29 19:13:20 EDT 2005
On Fri, Jul 29, 2005 at 05:00:31PM -0400, Adam Greene wrote:
> Hi,
>
> We have a customer who's experiencing high latency on IP traffic between New
> York (where we are) and their server in New Zealand. As far as I can tell,
> this is normal behavior.
You know traceroute is probably the second most dangerous tool on the
Internet in the hands of someone who doesn't know what to do with it,
right after those $#%^&*ing personal firewall programs that help people
detect and complain about their web browsing traffic and DNS requests.
> As far as I can tell, latency jumps on the transition from UUNet to Global
> Crossing, then again at the leap from Global Crossing to Asia Netcom. Can
> anyone tell from experience if this is normal behavior? I assume the GBLX /
> asianetcom transition is transatlantic fiber ...
Trans-pacific.
So the bottom line is that you're confused by the lack of DNS on hop 11
and the transition to a higher latency on hop 12? Lets review:
> Here's a snip from the traceroute:
>
> 7 37ms 1/ 100 = 1% 1/ 100 = 1% 500.POS4-2.GW1.NYC9.ALTER.NET
> [208.192.182.125]
This is your providers' customer edge router. This is what is known as a
customer demarc, usually a /30 on a point to point circuit (which this is,
a PoS interface), and the other side of this /30 is your router:
Name: hvdata-gw.customer.alter.net
Address: 208.192.182.126
Note that you start out coming in to this with 37ms of rtt latency, which
either means that the other end of this SONET circuit is really far away,
that the circuit is congested, or that someone is running out of CPU to
process TTL expired responses.
> 0/ 100 = 0% |
> 8 37ms 2/ 100 = 2% 2/ 100 = 2% 0.so-3-0-0.XL1.NYC9.ALTER.NET
> [152.63.99.178]
> 0/ 100 = 0% |
> 9 38ms 0/ 100 = 0% 0/ 100 = 0% 0.so-7-0-0.XL1.NYC8.ALTER.NET
> [152.63.0.37]
Even without knowing anything about UU's architecture, you can tell that
you are still in the same region, just a different building or POP, and
since the latency is staying consistant you know that it isn't a router
CPU exhaustion problem. Either you have a congested local pipe, or the
other end of this circuit runs somewhere far away.
> 0/ 100 = 0% |
> 10 38ms 0/ 100 = 0% 0/ 100 = 0% POS6-0.BR3.NYC8.ALTER.NET
> [152.63.19.53]
Now we're at a border router. One of the nice things about networks at UU
is that they are very structured, you can pretty much be guarantee that
customers won't be turning up on BR's and peers won't be turning up on
GW's etc. That means the next hop is probably going to be another network.
> 0/ 100 = 0% |
> 11 41ms 0/ 100 = 0% 0/ 100 = 0% 204.255.168.62
> 0/ 100 = 0% |
Ah and it is, but you have no way to know that. It seems that either
through policy or laziness someone has neglected to put reverse DNS on
the remote side of this /30. Of course you can always go looking at the
other side for more info:
Name: POS1-1.BR3.NYC8.ALTER.NET
Address: 204.255.168.61
You can also see that the /30 came out of UUNet's IP space, 204.252.0.0/14
to be precise. Not immediately helpful in figuring out who the other side
is, but it is 100% clear that this is a peer.
> 12 113ms 3/ 100 = 3% 3/ 100 = 3%
> so1-1-0-2488M.ar1.SJC2.gblx.net [67.17.64.65]
Ok well now you know who the other side of that last /30 was, Global
Crossing. The reason that you don't see any routing infrastructure inside
GBLX's network is because it is hidden within an MPLS LSP that is set to
not decrement the TTL as the label is switched through the network.
However, you can now tell that you are in SJC. A quick word about parsing
location identifiers. The most common types out there are:
1) Airport codes (like SJC)
2) CLLI geographic+geopolitical codes (like sjnsca)
3) Pseudo-airport codes (like NYC instead of LGA or JFK)
4) Random garbage from deranged minds
Luckily enough for you, GX and UU both fall into the category of "mostly"
real airport codes, with some fake ones thrown in for good measure just to
annoy people who don't know what to look for.
SJC = San Jose California, a real airport code
NYC = New York NY, a fake airport code, but one that is easy enough to
figure out.
This is crossing North America. By subjecting the previous latency of 41ms
from 113ms you can see that this hop has added 72ms of extra latency.
Perfectly normal for going that distance.
> 0/ 100 = 0% |
> 13 110ms 2/ 100 = 2% 2/ 100 = 2%
> ANC-San-Jose-3.ge-2-3-1.ar1.SJC2.gblx.net [64.215.184.246]
Hey look they confirmed it for you. Since you know that UU-GX is a peering
relationship, and you are UU's customer, you can pretty much infer that
ANC is a customer. This specific hop that you're looking at is another
customer demarc, between GX and ANC. You can see that the other side of
the /30 is:
Name: ge-2-3-1.ar1.SJC2.gblx.net
Address: 64.215.184.245
> 0/ 100 = 0% |
> 14 262ms 0/ 100 = 0% 0/ 100 = 0% po3-0.gw1.akl1.asianetcom.net
> [202.147.55.213]
So, now you know that ANC is Asia NetCom. You also know that the last
router in San Jose was theirs, so you now know that you have traveled
262ms - 110ms = 152ms on their network.
> 0/ 100 = 0% |
> 15 237ms 2/ 100 = 2% 2/ 100 = 2% 61.14.140.83
> 0/ 100 = 0% |
> 16 265ms 0/ 100 = 0% 0/ 100 = 0% vl5.cr1.akl.tranzpeer.net
> [202.180.64.118]
Yada yada yada.
Note that this entire process misses one of the most important elements of
deciphering a route, the reverse path. Of course since I don't have that
data, and it appears to be irrelevent anyways, I'm just going to ignore
that. Answer is, perfectly normal latency.
> Just trying to find something to tell the customer short of hosting the
> server here in NY.
The fiber paths COULD be a tiny bit better. 72ms is actually on the high
side for this route, a better route could come out somewhere in the 66ms
range. What you're actually seeing is 76ms on the GX NYC->SJC rtt and
something lower on the UU path of SJC->NYC, which balances out to 72ms
when you consider both sides of the path. If you're a GX customer you
probably understand why that is, but that is a discussion for another day.
:)
The best you could hope for even with perfectly optimized fiber paths is
only a few ms of improvement. If you want to make it significantly better,
either move one of the endpoints, or make light move through fiber faster.
:)
--
Richard A Steenbergen <ras at e-gerbil.net> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
More information about the cisco-nsp
mailing list