[outages] Cox -> nLayer connectivity issues

Cary Wiedemann carywiedemann at gmail.com
Wed Dec 19 19:17:53 EST 2012


All,

I knew I should have checked this list before opening tickets far and
wide.  I've been experiencing this issue since before 5:30pm EST and just
wanted to report that ICMP *IS* affected for me, but only for certain IP
addresses.  TCP seems to be intermittently affected.

I host a server at InfoRelay with network 69.169.88.16/28.  From a Cox
Communications optical internet circuit I can ping 69.169.88.20 and .21,
but not .22 .23 or .24.

chantilly-asa# ping 69.169.88.20
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 69.169.88.20, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

chantilly-asa# ping 69.169.88.21
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 69.169.88.21, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

chantilly-asa# ping 69.169.88.22
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 69.169.88.22, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)

chantilly-asa# ping 69.169.88.23
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 69.169.88.23, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)

A good trace looks like this (first hop obscured):

C:\>tracert 69.169.88.20

Tracing route to schneller.carywiedemann.com [69.169.88.20] over a maximum
of 30 hops:

  1     1 ms     1 ms     1 ms
wsip-174-000-000-000-dc.dc.cox.net[174.000.000.000]
  2     1 ms     1 ms     1 ms  mrfddsrj01gex070003.rd.dc.cox.net[68.100.0.141]
  3     2 ms     2 ms     2 ms  68.1.4.139
  4    11 ms     6 ms     4 ms  xe-5-0-7.ar1.iad1.us.nlayer.net[69.31.10.81]
  5   203 ms   204 ms   209 ms
as33597.xe-3-0-5-304.ar1.iad1.us.nlayer.net[69.31.10.70]
  6     3 ms     3 ms     3 ms  cr1.iad1.inforelay.net [66.231.176.9]
  7     4 ms     5 ms     3 ms  cr1.iad4.inforelay.net [66.231.177.66]
  8     3 ms     3 ms     3 ms  schneller.carywiedemann.com [69.169.88.20]

While a trace to 69.169.88.22 dies after hop 3:
C:\>tracert 69.169.88.22

Tracing route to fairfaxunderground.com [69.169.88.22]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms
wsip-174-000-000-000.dc.dc.cox.net[174.000.000.000]
  2     1 ms     1 ms     1 ms  mrfddsrj01gex070003.rd.dc.cox.net[68.100.0.141]
  3     *        2 ms     2 ms  68.1.4.139
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.

Although TCP connections still work, they're highly intermittent.  I
haven't had a successful ICMP echo reply from 69.169.88.22 or 69.169.88.23
from a Cox connection via nLayer for several hours.

I'm asking both Cox and InfoRelay to depeer from nLayer.

Feel free to use my server as an ICMP target.

- Cary

On Wed, Dec 19, 2012 at 6:53 PM, Corey Quinn <corey at sequestered.net> wrote:

> Also in LA here.
>
> traceroute to arin.net (192.149.252.76), 30 hops max, 60 byte packets
>  1  10.201.69.1 (10.201.69.1)  0.227 ms  0.233 ms  0.232 ms
>  2  * * *
>  3  xe-7-2-0.mpr1.lax112.us.above.net (64.125.170.97)  1.424 ms  1.396 ms
>  1.366 ms
>  4  above-cox-1.lax12.us.above.net (64.125.13.10)  1.331 ms above-cox-2.
> lax12.us.above.net (64.125.13.14)  1.411 ms above-cox-1.lax12.us.above.net(64.125.13.10)  1.386 ms
>  5  mrfddsrj02-ae0.0.rd.dc.cox.net (68.1.1.7)  67.585 ms mrfddsrj01-ae0.0.
> rd.dc.cox.net (68.1.1.5)  67.667 ms  67.783 ms
>  6  * * *
>  7  * * *
>  8  wsip-98-172-152-14.dc.dc.cox.net (98.172.152.14)  79.017 ms  69.115
> ms  69.117 ms
>  9  * * *
> 10  * * *
> 11  * * *
>
>
>
> On Dec 19, 2012, at 3:50 PM, Jake Mertel <jake at nobistech.net> wrote:
>
> Something else that just clicked, I have been having a number of issues
> reaching arin.net today from one of my servers in Los Angeles that uses
> nLayer as its upstream. Request response times are between 20 and 40
> seconds as opposed to 2 to 4 seconds on our office connection. Looking at
> my trace from LA, we are going LA<->Cox<->ARIN.****
>
> C:\Users\jake>tracert arin.net****
>
> Tracing route to arin.net [192.149.252.76]****
> over a maximum of 30 hops:****
>
>   1    <1 ms     1 ms    <1 ms  v403.er01.lax.ubiquity.io [72.37.224.129]*
> ***
>   2     1 ms     5 ms     1 ms  xe-1-0-3.ar1.lax2.us.nlayer.net
> [69.31.127.45]****
>   3    <1 ms    <1 ms    <1 ms  ae1-80g.cr1.lax1.us.nlayer.net
> [69.31.127.129]****
>   4     2 ms     5 ms     2 ms  ae2-50g.ar1.lax1.us.nlayer.net
> [69.31.127.142]****
>   5    <1 ms    <1 ms    <1 ms  as22773.ae12.ar1.lax1.us.nlayer.net
> [69.31.127.230]****
>   6    70 ms   111 ms    70 ms  mrfddsrj01-ae0.0.rd.dc.cox.net [68.1.1.5]*
> ***
>   7     *        *        *     Request timed out.****
>   8     *        *        *     Request timed out.****
>   9    72 ms    73 ms    82 ms  wsip-98-172-152-14.dc.dc.cox.net
> [98.172.152.14]****
> 10    72 ms    72 ms    82 ms  host-252-131.arin.net [192.149.252.131]****
> 11     *        *        *     Request timed out.****
> 12     *        *        *     Request timed out.****
> 13     *        *        *     Request timed out.****
> 14     *        *        *     Request timed out.****
> 15     *        *        *     Request timed out.****
>
>
> *From:* outages-bounces at outages.org [mailto:outages-bounces at outages.org] *On
> Behalf Of *Jake Mertel
> *Sent:* Wednesday, December 19, 2012 4:45 PM
> *To:* 'Brandon Whaley'; 'outages at outages.org'
> *Subject:* Re: [outages] Cox -> nLayer connectivity issues****
> ** **
> We have received a report of similar issues today. The client has servers
> with us in several locations where we use nLayer and/or PacketExchagne and
> his monitoring system is on a network that uses Cox as its preferred
> upstream. He shutdown his Cox upstream and didn’t have any issues reaching
> the servers over his backup provider. The issues were sporadic and did not
> affect all protocols – ICMP pings worked, snmpwalk was fine, but UDP traces
> were dying somewhere on the reverse path. Seems to be very similar to what
> you are seeing.****
>
> *From:* outages-bounces at outages.org [mailto:outages-bounces at outages.org<outages-bounces at outages.org>
> ] *On Behalf Of *Brandon Whaley
> *Sent:* Wednesday, December 19, 2012 4:25 PM
> *To:* outages at outages.org
> *Subject:* [outages] Cox -> nLayer connectivity issues****
> ** **
> We've been seeing intermittent TCP/UDP connectivity issues from Cox
> Communications in Virginia to any location that routes over nLayer.  UDP
> traceroutes are fine, but DNS lookups time out for minutes at a time, then
> work again for ~5 minutes before repeating the problem.  ICMP is never
> affected during the outages.****
> ** **
> traceroute to 198.46.80.1 (198.46.80.1), 30 hops max, 60 byte packets****
>  1  router36f24c.local (192.168.14.1)  0.560 ms  0.531 ms  0.753 ms****
>  2  wsip-174-77-92-169.hr.hr.cox.net (174.77.92.169)  2.532 ms  2.581 ms
>  3.009 ms****
>  3  172.21.224.153 (172.21.224.153)  3.995 ms  4.067 ms  4.111 ms****
>  4  172.21.249.101 (172.21.249.101)  4.185 ms  4.396 ms  4.561 ms****
>  5  172.21.249.73 (172.21.249.73)  4.916 ms  4.900 ms  5.128 ms****
>  6  172.21.249.18 (172.21.249.18)  5.517 ms  5.124 ms  5.043 ms****
>  7  ip-216-54-33-22.coxfiber.net (216.54.33.22)  210.486 ms  210.477 ms
>  210.466 ms****
>  8  68.1.4.139 (68.1.4.139)  221.950 ms  222.460 ms  232.268 ms****
>  9  * xe-5-0-7.ar1.iad1.us.nlayer.net (69.31.10.81)  226.332 ms  226.866
> ms****
> 10  as54641.xe-9-0-1.ar1.iad1.us.nlayer.net (69.31.31.42)  223.718 ms
>  225.083 ms  225.744 ms****
> 11  198.46.80.1 (198.46.80.1)  225.706 ms  226.670 ms  226.698 ms****
> ** **
> Is anyone with Cox on the list that can investigate/contact me?****
> ** **
> --
> Best Regards,
> Brandon W.****
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20121219/8c93220d/attachment.htm>


More information about the Outages mailing list