[outages] Internap/XO/Savvis problems?

[Matt Palmer]

Greetings Outageers,

One of our customers is seeing some problems between themselves and our DC,
and both-way traceroutes are showing problems just corewards of our
upstream, Internap:

Customer to us:

traceroute to 74.201.255.113 (74.201.255.113), 64 hops max, 40 byte packets
 1  192.168.1.1 (192.168.1.1)  3.188 ms  7.275 ms  5.663 ms
 2  ATM1-0-60.win17.Melbourne.telstra.net (165.228.176.49)  12.096 ms  9.410 ms  13.803 ms
 3  TenGigE0-8-0-5.win-core1.Melbourne.telstra.net (203.50.80.129)  8.397 ms  8.633 ms  11.843 ms
 4  Bundle-Pos1.ken-core4.Sydney.telstra.net (203.50.6.21)  22.303 ms  26.382 ms  29.742 ms
 5  Port-Channel1.pad-gw2.Sydney.telstra.net (203.50.6.29)  21.379 ms  26.954 ms  31.464 ms
 6  10GigabitEthernet1-0.sydp-core02.Sydney.reach.com (203.50.13.46)  115.516 ms  24.861 ms  204.107 ms
 7  i-0-0.wil-core02.net.reach.com (202.84.144.101)  167.032 ms  170.378 ms  175.605 ms
 8  p9_0-ashc1.net.reach.com (202.84.140.90)  182.026 ms  184.921 ms  181.750 ms
 9  g4_14-pax06.net.reach.com (202.84.251.106)  181.076 ms  183.144 ms  181.113 ms
10  206.111.12.173.ptr.us.xo.net (206.111.12.173)  192.088 ms  324.819 ms  194.243 ms
11  * * te-11-0-0.rar3.sanjose-ca.us.xo.net (207.88.12.69)  196.075 ms
12  207.88.14.133.ptr.us.xo.net (207.88.14.133)  261.502 ms  259.664 ms  266.501 ms
13  207.88.14.134.ptr.us.xo.net (207.88.14.134)  264.905 ms  257.116 ms  258.146 ms
(oh so many stars)

Us back to the customer:

traceroute to 165.228.176.49 (165.228.176.49), 30 hops max, 40 byte packets
 1  33.254.201.74.static.ey03.engineyard.com (74.201.254.33)  0.907 ms  0.908 ms  0.928 ms
 2  76.254.201.74.static.ey03.engineyard.com (74.201.254.76)  0.210 ms  0.241 ms  0.273 ms
 3  core1.te6-1-bbnet1.nym007.pnap.net (216.52.95.24)  1.559 ms core1.te6-2-bbnet2.nym007.pnap.net (216.52.95.88)  1.338 ms core1.te6-1-bbnet1.nym007.pnap.net (216.52.95.24)  1.369 ms
 4  208.173.129.61 (208.173.129.61)  1.441 ms  1.525 ms  1.658 ms
 5  * * *
 6  cr2-loopback.sfo.savvis.net (206.24.210.71)  98.457 ms  94.617 ms  101.634 ms
 7  pr1-so-0-0-0.PaloAltoPaix.savvis.net (204.70.200.193)  89.389 ms  89.397 ms  89.478 ms
(constellations from here on)

I'm fine from my connection on the same side of the pond, since it looks
like my ISP has a direct peering with Internap:

traceroute to 74.201.255.113 (74.201.255.113), 30 hops max, 40 byte packets
 1  172.14.17.1 (172.14.17.1)  34.982 ms  35.987 ms  48.326 ms
 2  lns10.syd6.internode.on.net (150.101.197.88)  50.219 ms  51.812 ms  53.293 ms
 3  vl14.cor2.syd6.internode.on.net (150.101.197.83)  91.320 ms  92.807 ms  94.274 ms
 4  po3.bdr1.syd6.internode.on.net (150.101.199.232)  226.619 ms  231.256 ms  234.897 ms
 5  pos2-1.bdr1.lax1.internode.on.net (203.16.213.158)  228.912 ms  233.132 ms  236.411 ms
 6  cr1-eqix-peer.lax009.internap.net (206.223.123.29)  245.254 ms  208.194 ms *
 7  cr1.wdc005.inappnet-65.cr1.lax009.internap.net (66.79.146.141)  279.374 ms  283.716 ms  289.151 ms
 8  cr1.nym009.inappnet-84.cr1.wdc005.internap.net (66.79.146.209)  309.854 ms  316.382 ms  318.303 ms
 9  core3.nym.inappnet-82.cr1.nym007.internap.net (66.79.152.130)  298.775 ms  300.648 ms  307.357 ms
10  border2.pc2-bbnet2.nyj001.pnap.net (216.52.95.74)  323.780 ms  325.643 ms  331.222 ms
11  74.201.254.73 (74.201.254.73)  348.343 ms  350.727 ms *
12  113.255.201.74.static.ey03.engineyard.com (74.201.255.113)  305.041 ms  303.767 ms  305.289 ms

Wondering if anyone else is seeing similar problems, or if anyone knows of
any widespread problems with Internap/XO/Savvis that I can pass on to my
customer to let them know what's going on.

Thanks,
- Matt