[outages] Zayo issues in denver?
chris
tknchris at gmail.com
Fri Dec 14 00:36:07 EST 2012
Yeah I have a ticket open with softlayer as it seems they are the most
common factor and the only one i have a direct relationship in this
situation, no response so far in about an hour but hopefully before
the workforce starts ringing phones off the hook :)
On Fri, Dec 14, 2012 at 12:28 AM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> For Dallas->Denver, the latency begins between hop #7 (360.net in
> LAX/Los Angeles) and hop #8 (360.net in DEN/Denver). The latency
> at hop #6 is probably the result of ICMP prioritisation (indicated by
> lower latency at hop #7).
>
> For Denver->Dallas, latency and packet loss begins between hop #4
> (Level3 in Denver) and hop #5 (SoftLayer in Denver).
>
> Thus, to solve this issue, you would need to have the following 3
> providers all on the phone at the same time talking to one another
> simultaneously, and give all 3 of them the below traceroutes/mtrs with
> evidence (because they need to know what physical interfaces the packets
> are flowing across):
>
> - 360.net / Zayo / Abovenet
> - Level3
> - SoftLayer
>
> The most you can do yourself is contact the providers who you have
> service with (i.e. SoftLayer and GreenhouseData.net(?)) and ask if there
> is anything they can do or contact their peering providers. Contact
> them and put pressure on them.
>
> Because the issue is so far "in" (i.e. not at hop #2 or #3 within
> traceroutes), there is very little you can do. Even if you had peering
> arrangements with, say, L3 and 360/Zayo/AboveNet, its not evident from
> the traceroutes and you'll usually be ignored (or more specifically you
> will be asked repeatedly "What's your customer ID?" and if you don't
> have one, basically told to sod off).
>
> Remember: the Internet is always broken, 24x7x365, in some regard. It
> sucks when your packets suffer from it, but there's very little you can
> do when you don't have contractual arrangements with the providers who
> are experiencing issues.
>
> P.S. -- From my home Comcast connection in Mountain View, I see no
> issues to either 208.89.160.11 nor 70.87.254.1 -- but my packets go
> through completely different interfaces and/or routers than whats shown
> in your traceroutes. The same goes from tests performed from my VPS in
> Los Angeles (who has peering with SoftLayer directly) -- again, packets
> going through completely different interfaces.
>
> --
> | Jeremy Chadwick jdc at koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Mountain View, CA, US |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>
> On Fri, Dec 14, 2012 at 12:11:45AM -0500, chris wrote:
>> Ok sorry, the denver side for us is 208.89.160.11 and the dallas
>> softlayer side for us is 70.87.254.1
>>
>> You are right about different paths it appears that the route back is
>> direct via level3 instead of 360/zayo and same latency is present so
>> looks like more likely a softlayer issue?
>>
>> trace from our dallas box(softlayer) to our denver box(greenhouse):
>> 2: po101.dsr01.dllstx5.networklayer.com (70.87.254.1) 1.941ms
>> 3: po51.dsr01.dllstx3.networklayer.com (70.85.127.105) 2.217ms
>> 4: ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224) 1.719ms
>> 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 32.448ms
>> 6: 360.net.any2ix.coresite.com (206.223.143.201) 657.255ms
>> 7: lax1-core-01-xe-0-0-0.360.net (66.62.2.213) 36.196ms
>> 8: den1-core-01-ae0.360.net (66.62.2.169) 246.045ms
>> 9: den1-edge-01-lag2.360.net (66.62.2.194) asymm 8 271.648ms
>> 10: 66.62.160.30 (66.62.160.30) asymm 9 543.816ms
>> 11: CYSWYDC01ESW1-001-1-1.GREENHOUSEDATA.NET (208.89.160.11) asymm 10
>> 793.839ms reached
>>
>> from denver box to dallas (mtr is all thats installed and cant install
>> anything else connectivity is too horrible)
>>
>>
>> Packets Pings
>> Host
>>
>> Loss% Snt Last Avg Best Wrst StDev
>>
>> 2. CYSWYDC01RTR1-001-0-1.GREENHOUSEDATA.NET > > 0.0% 17 0.3 0.3 0.3 0.4 0.0
>> 3. ge-6-13.car2.Denver1.Level3.net > > 5.9% 17 4.6 121.9 4.6 417.4 116.1
>> 4. ae-21-52.car1.Denver1.Level3.net > > 0.0% 17 101.9 25.2 2.9 140.6 40.6
>> 5. te1-5.bbr01.cf01.den01.networklayer.com > > 43.8% 17 565.9 479.9 169.5 919.2 214.1
>> 6. ae7.bbr01.cs01.den01.networklayer.com > > 37.5% 17 642.3 665.4 170.0 928.8 261.4
>> 7. ae12.bbr02.eq01.dal03.networklayer.com > > 50.0% 17 1168. 650.2 95.4 1168. 391.4
>> 8. po32.dsr01.dllstx3.networklayer.com > > 31.2% 17 423.9 643.3 80.0 1135. 368.2
>> 9. po101.dsr01.dllstx5.networklayer.com > > 50.0% 16 347.1 814.1 347.1 1874. 508.6
>>
>> thanks for pointing it out
>> chris
>>
>> On Fri, Dec 14, 2012 at 12:00 AM, Jeremy Chadwick <jdc at koitsu.org> wrote:
>> >
>> > Chris,
>> >
>> > You need to provide traceroutes from both directions. The below is for
>> > Dallas --> Denver, but the return path (Denver --> Dallas) needs to be
>> > provided too. You might find that the return path goes through some
>> > provider other than Zayo/360/Abovenet/whateverthey'recalledtoday.
>> >
>> > I know that's hard to do when the path has latency or packet loss (which
>> > you don't show in your results -- you only show latency), but this is
>> > exactly what a cronjob traceroute writing to a log file is for. :-)
>> >
>> > My point: remember that routing on the Internet most of the time is
>> > asymmetric. Reference material (read, do not skim):
>> >
>> > http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf
>> >
>> > Finally, you didn't provide IP addresses of either server (in Denver or
>> > Dallas), so when you ask for "someone else to test", that's not easily
>> > doable, at least not to the endpoints involved (pinging routers is not
>> > sufficient evidence, sadly).
>> >
>> > --
>> > | Jeremy Chadwick jdc at koitsu.org |
>> > | UNIX Systems Administrator http://jdc.koitsu.org/ |
>> > | Mountain View, CA, US |
>> > | Making life hard for others since 1977. PGP 4BD6C0CB |
>> >
>> > On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
>> > > Seeing massive packet loss and high latency from 360.net (which appears to
>> > > be owned by zayo/abovenet/whatever) in denver....
>> > >
>> > > Here is a trace snip from our server @ softlayer in dallas to one of our
>> > > servers in denver
>> > >
>> > > 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms
>> > > 6: no reply
>> > > 7: 66.62.2.213 (66.62.2.213) 4588.320ms
>> > > 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
>> > >
>> > > I only have one one server in denver that I can reproduce this with hoping
>> > > someone can else can test and reproduce similar issues
>> > >
>> > > thanks
>> > > chris
>> >
>> > > _______________________________________________
>> > > Outages mailing list
>> > > Outages at outages.org
>> > > https://puck.nether.net/mailman/listinfo/outages
>> >
More information about the Outages
mailing list