[outages] Level3 Routing...
Jeremy Chadwick
jdc at koitsu.org
Fri Feb 28 20:49:50 EST 2014
I'd suggest setting up a cron job that logs the mtr data on two systems
(system A, going from A->B, and system B, going from B->A), so that you
can review it if/when it happens again.
I have a small shell script I wrote, which uses a dorky config file of
sorts, that can do this for you if you want me to provide it. I use it
on my home FreeBSD system, and my FreeBSD VPS system, checking
connectivity between them (and to some other places as well).
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |
On Sat, Mar 01, 2014 at 12:25:43AM +0000, jay lark wrote:
> Correct, no packet loss in that mtr but packet loss is occurring from time to time. The main point of that particular trace was to show the latency which is not normal. That same trace now is reporting times around 165ms. The reverse trace which appears to go back along the same path is around 160ms.
>
> Again the problem has been off and on for a while now as others here have said. I'm just adding my data point and have a ticket with level3 as well.
>
> -----Original Message-----
> From: Jeremy Chadwick [mailto:jdc at koitsu.org]
> Sent: Friday, February 28, 2014 5:55 PM
> To: jay lark
> Cc: 'Outages at outages.org'
> Subject: Re: [outages] Level3 Routing...
>
> I see no indication of packet loss in this mtr; others have already explained how to properly read output. Loss at hop #3 is not indicative of a universal problem; it would be different if that latency "trickled down" through all subsequent hops, all the way to the destination.
>
> I do see indication in this mtr of generally high latency (between London<->Paris), reaching up to 246ms and averaging around 171ms. I do not know if that's normal or not, due to my lack of familiarity with said region/paths.
>
> However, as I've stated repeatedly [like a broken record], a traceroute/mtr from point A to B is only half the picture -- you need the same from point B to A, due to asymmetrical routing being commonplace.
>
> --
> | Jeremy Chadwick jdc at koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>
> On Fri, Feb 28, 2014 at 10:04:18PM +0000, jay lark wrote:
> > We've also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our ATT office in Chicago have been okay for the most part.
> >
> > Here's a current trace from London to LAX. Heavy packet loss in
> > London. It was in Paris before that :)
> >
> > My traceroute [v0.80]
> > lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014
> > Resolver: Received error response 2. (server failure)er of fields quit
> > Packets Pings
> > Host Loss% Snt Last Avg Best Wrst StDev
> > 1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3
> > 2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7
> > 3. ae-0-11.edge3.london2.level3.net 61.8% 468 1.1 1.6 1.0 18.0 2.5
> > 4. ae-3-3.ebr1.paris1.level3.net 0.0% 468 109.9 171.8 103.0 246.0 26.9
> > 5. ae-91-91.csw4.paris1.level3.net 0.2% 468 111.7 170.4 99.6 241.8 27.0
> > 6. ae-92-92.ebr2.paris1.level3.net 0.0% 468 121.8 170.1 95.8 236.3 27.0
> > 7. ae-41-41.ebr2.washington1.level3.net 0.0% 467 118.5 171.4 95.7 241.1 27.1
> > 8. ae-45-45.ebr2.washington12.level3.net 0.0% 467 113.4 171.8 94.9 243.5 27.0
> > 9. ae-204-3604.edge1.washington12.level3.net 0.0% 467 107.7 171.7 95.4 250.7 27.8
> > ae-201-3601.edge1.washington12.level3.net
> > ae-203-3603.edge1.washington12.level3.net
> > ae-202-3602.edge1.washington12.level3.net
> > 10. 208.51.134.89 0.0% 467 144.0 233.5 124.8 454.1 47.8
> > 11. po1-20g.ar6.dca3.gblx.net 0.0% 467 148.8 226.8 125.8 448.9 37.7
> > 12. uunet-2.ar3.dca3.gblx.net 0.0% 467 146.1 225.9 126.5 319.5 32.9
> > 13. ???
> > 14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4
> > 15. whitehousepost-gw.customer.alter.net 0.0% 467 210.6 289.2 200.1 368.3 30.6
> >
> >
> > From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
> > alexzarris at yahoo.com
> > Sent: Tuesday, February 25, 2014 11:00 AM
> > To: 'Clayton Zekelman'; 'Michael Morrison'; 'Outages at outages.org'
> > Subject: Re: [outages] Level3 Routing...
> >
> >
> > We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes:
> >
> >
> >
> > "Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral."
> >
> >
> >
> > However, we're still having internet issues.
> >
> > Alex
> >
> > From: Outages [mailto:outages-bounces at outages.org] On Behalf Of
> > Clayton Zekelman
> > Sent: Tuesday, February 25, 2014 7:04 AM
> > To: Michael Morrison; Outages at outages.org<mailto:Outages at outages.org>
> > Subject: Re: [outages] Level3 Routing...
> >
> >
> > That's what we told our customer... by the time they replied, the issue had gone away.
> >
> > At 09:57 AM 25/02/2014, Michael Morrison wrote:
> > Content-Language: en-US
> > Content-Type: multipart/alternative;
> > boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"
> >
> > As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.
> >
> > From: Outages [ mailto:outages-bounces at outages.org] On Behalf Of
> > Clayton Zekelman
> > Sent: Tuesday, February 25, 2014 6:39 AM
> > To: *Network; Luke Rockwell; Petter Bruland
> > Cc: Outages at outages.org<mailto:Outages at outages.org>
> > Subject: Re: [outages] Level3 Routing...
> >
> >
> >
> > We had a report from a customer earlier this morning, but they advised the problem went away.
> >
> > We connect to Level3 at Southfield MI.
> >
> > At 09:35 AM 25/02/2014, *Network wrote:
> > Content-Language: en-US
> > Content-Type: multipart/alternative;
> > boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_"
> > Latency issue with Level 3 here in Detroit area
> >
> > Anton
> >
> >
> >
> > From: Outages [ <mailto:outages-bounces at outages.org>
> > mailto:outages-bounces at outages.org] On Behalf Of Luke Rockwell
> > Sent: Tuesday, February 18, 2014 12:24 PM
> > To: Petter Bruland
> > Cc: Outages at outages.org<mailto:Outages at outages.org>
> > Subject: Re: [outages] Level3 Routing...
> >
> > nothing on the level 3 network events page I have access to Luke
> >
> > On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland < Petter.Bruland at allegiantair.com<mailto:Petter.Bruland at allegiantair.com>> wrote:
> > Not seeing any issues with Level3 in Las Vegas.
> >
> > -Petter
> >
> > From: Outages [ <mailto:outages-bounces at outages.org>
> > mailto:outages-bounces at outages.org] On Behalf Of Nick Olsen
> > Sent: Tuesday, February 18, 2014 6:04 AM
> > To: Morgan Miskell; Outages at outages.org<mailto:Outages at outages.org>
> > Subject: Re: [outages] Level3 Routing...
> >
> > Not seeing anything homed off of Level 3 Orlando.
> >
> > Traceroutes to a few common places look fine.
> >
> > ~474815 Routes from 3356.
> > Nick Olsen
> > Network Operations
> > (855) FLSPEED x106
> >
> > ________________________________
> > From: "Morgan Miskell"
> > <morgan.miskell at caro.net<mailto:morgan.miskell at caro.net> >
> > Sent: Tuesday, February 18, 2014 9:00 AM
> > To: "Outages at outages.org<mailto:Outages at outages.org>"
> > <outages at outages.org<mailto:outages at outages.org>>
> > Subject: [outages] Level3 Routing...
> > Anyone seeing Level3 routing issues? We just observed a massive drop
> > in routes in the BGP feeds and seeing some strange ICMP issues along
> > their backbone in terms of traceroutes along their backbone. They
> > might be adding some filters based on the recent DDOS attacks, I can
> > ping interfaces but traceroutes go to hell...
> > --
> > Morgan A. Miskell
> > CaroNet Data Centers
> > 704-643-8330 x206
> > ----------------------------------------------------------------------
> > ------ The information contained in this e-mail is confidential and is
> > intended only for the named recipient(s). If you are not the intended
> > recipient you must not copy, distribute, or take any action or
> > reliance on it. If you have received this e-mail in error, please
> > notify the sender. Any unauthorized disclosure of the information
> > contained in this e-mail is strictly prohibited.
> > ----------------------------------------------------------------------
> > ------ _______________________________________________
> > Outages mailing list
> > Outages at outages.org<mailto:Outages at outages.org>
> > https://puck.nether.net/mailman/listinfo/outages
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org<mailto:Outages at outages.org>
> > https://puck.nether.net/mailman/listinfo/outages
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org<mailto:Outages at outages.org>
> > https://puck.nether.net/mailman/listinfo/outages
> > ---
> >
> > Clayton Zekelman
> > Managed Network Systems Inc. (MNSi)
> > 3363 Tecumseh Rd. E
> > Windsor, Ontario
> > N8W 1H4
> >
> > tel. 519-985-8410
> > fax. 519-985-8409
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org<mailto:Outages at outages.org>
> > https://puck.nether.net/mailman/listinfo/outages
> >
> > ---
> >
> > Clayton Zekelman
> > Managed Network Systems Inc. (MNSi)
> > 3363 Tecumseh Rd. E
> > Windsor, Ontario
> > N8W 1H4
> >
> > tel. 519-985-8410
> > fax. 519-985-8409
>
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
More information about the Outages
mailing list