[outages] Zayo issues in denver?
chris
tknchris at gmail.com
Fri Dec 14 01:08:39 EST 2012
Hi Tim,
Just tracerouted your ip and saw a hop in the same location:
11. xe-0-0-1.mpr1.den1.above.net 0.0% 32 59.6 60.0 59.3 70.6 1.9
12. 208.185.20.58.t01657-09.above.ne 0.0% 32 59.7 60.3 59.4 76.0 2.9
13. den1-edge-01-lag2.360.net 46.9% 32 60.6 63.0 60.3 95.1 8.3
14. 66.62.160.42 0.0% 32 200.1 79.1 65.9 200.1 37.1
15. static-65-19-63-33.cybermesa.com 0.0% 32 248.4 80.9 67.8 248.4 41.3
Hop 13 was one of the links we were hitting and seeing alot of latency
and loss, also noticed its acronym LAG maybe this was some kind of
weird LACP failure
Anyway after checking your side and seeing it was normal, I retested
and the issue appears to be resolved for me in both directions
So either someone on here silently bumped the issue and got it fixed
or softlayer was able to resolve this after my ticket
Either way, thanks!
chris
On Fri, Dec 14, 2012 at 12:41 AM, Tim Densmore
<tdensmore at tarpit.cybermesa.com> wrote:
> Hi Chris,
>
> Not entirely what we saw tonight, but we had issues reaching some places via
> a very similar path, and we peer with 360 and level3. I've localprefed
> around it for the time being, but only just saw this email. Can't confirm
> our issue was what you were seeing, but we were definitely seeing packet
> loss, latency, and what appeared to be occasional loops on networklayer.
> Not particularly useful, I realise, but I guess count me in as a tentative
> "me too."
>
>
> TD
>
>
>
> On 12/13/2012 10:11 PM, chris wrote:
>>
>> Ok sorry, the denver side for us is 208.89.160.11 and the dallas
>> softlayer side for us is 70.87.254.1
>>
>> You are right about different paths it appears that the route back is
>> direct via level3 instead of 360/zayo and same latency is present so
>> looks like more likely a softlayer issue?
>>
>> trace from our dallas box(softlayer) to our denver box(greenhouse):
>> 2: po101.dsr01.dllstx5.networklayer.com (70.87.254.1) 1.941ms
>> 3: po51.dsr01.dllstx3.networklayer.com (70.85.127.105) 2.217ms
>> 4: ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224) 1.719ms
>> 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 32.448ms
>> 6: 360.net.any2ix.coresite.com (206.223.143.201) 657.255ms
>> 7: lax1-core-01-xe-0-0-0.360.net (66.62.2.213) 36.196ms
>> 8: den1-core-01-ae0.360.net (66.62.2.169) 246.045ms
>> 9: den1-edge-01-lag2.360.net (66.62.2.194) asymm 8
>> 271.648ms
>> 10: 66.62.160.30 (66.62.160.30) asymm 9
>> 543.816ms
>> 11: CYSWYDC01ESW1-001-1-1.GREENHOUSEDATA.NET (208.89.160.11) asymm 10
>> 793.839ms reached
>>
>> from denver box to dallas (mtr is all thats installed and cant install
>> anything else connectivity is too horrible)
>>
>>
>> Packets Pings
>> Host
>>
>> Loss% Snt Last Avg Best Wrst StDev
>>
>> 2. CYSWYDC01RTR1-001-0-1.GREENHOUSEDATA.NET
>>
>> 0.0% 17 0.3 0.3 0.3 0.4 0.0
>> 3. ge-6-13.car2.Denver1.Level3.net
>>
>> 5.9% 17 4.6 121.9 4.6 417.4 116.1
>> 4. ae-21-52.car1.Denver1.Level3.net
>>
>> 0.0% 17 101.9 25.2 2.9 140.6 40.6
>> 5. te1-5.bbr01.cf01.den01.networklayer.com
>>
>> 43.8% 17 565.9 479.9 169.5 919.2 214.1
>> 6. ae7.bbr01.cs01.den01.networklayer.com
>>
>> 37.5% 17 642.3 665.4 170.0 928.8 261.4
>> 7. ae12.bbr02.eq01.dal03.networklayer.com
>>
>> 50.0% 17 1168. 650.2 95.4 1168. 391.4
>> 8. po32.dsr01.dllstx3.networklayer.com
>>
>> 31.2% 17 423.9 643.3 80.0 1135. 368.2
>> 9. po101.dsr01.dllstx5.networklayer.com
>>
>> 50.0% 16 347.1 814.1 347.1 1874. 508.6
>>
>> thanks for pointing it out
>> chris
>>
>> On Fri, Dec 14, 2012 at 12:00 AM, Jeremy Chadwick <jdc at koitsu.org> wrote:
>>>
>>> Chris,
>>>
>>> You need to provide traceroutes from both directions. The below is for
>>> Dallas --> Denver, but the return path (Denver --> Dallas) needs to be
>>> provided too. You might find that the return path goes through some
>>> provider other than Zayo/360/Abovenet/whateverthey'recalledtoday.
>>>
>>> I know that's hard to do when the path has latency or packet loss (which
>>> you don't show in your results -- you only show latency), but this is
>>> exactly what a cronjob traceroute writing to a log file is for. :-)
>>>
>>> My point: remember that routing on the Internet most of the time is
>>> asymmetric. Reference material (read, do not skim):
>>>
>>>
>>> http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf
>>>
>>> Finally, you didn't provide IP addresses of either server (in Denver or
>>> Dallas), so when you ask for "someone else to test", that's not easily
>>> doable, at least not to the endpoints involved (pinging routers is not
>>> sufficient evidence, sadly).
>>>
>>> --
>>> | Jeremy Chadwick jdc at koitsu.org |
>>> | UNIX Systems Administrator http://jdc.koitsu.org/ |
>>> | Mountain View, CA, US |
>>> | Making life hard for others since 1977. PGP 4BD6C0CB |
>>>
>>> On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
>>>>
>>>> Seeing massive packet loss and high latency from 360.net (which appears
>>>> to
>>>> be owned by zayo/abovenet/whatever) in denver....
>>>>
>>>> Here is a trace snip from our server @ softlayer in dallas to one of our
>>>> servers in denver
>>>>
>>>> 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms
>>>> 6: no reply
>>>> 7: 66.62.2.213 (66.62.2.213) 4588.320ms
>>>> 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
>>>>
>>>> I only have one one server in denver that I can reproduce this with
>>>> hoping
>>>> someone can else can test and reproduce similar issues
>>>>
>>>> thanks
>>>> chris
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
More information about the Outages
mailing list