[f-nsp] Router Becomes Unresponsive

Wed Mar 11 18:42:24 EDT 2015

In a stick configuration make sure 'no route-only' is entered if you are using VLANs w/ VEs. This one used to get me all the time because default is route-only.

Also have your cam-partition profile and system-maxes setup for 1mil routes.

----- Original Message -----
From: "Chris O'Brien" <chris.obrien.1021 at gmail.com>
To: "Jake Mertel" <jake at nobistech.net>, foundry-nsp at puck.nether.net
Sent: Wednesday, March 11, 2015 12:58:01 PM
Subject: Re: [f-nsp] Router Becomes Unresponsive

Jake, 

Comments inline. Thanks in advance for your help. 

"1) When the issue is occurring, what are you seeing in 'show tasks' and 'show cpu lp'?" 

CPU was showing 100% idle. 

1A) Is the BGP task what is taking up your MP CPU in 'show tasks'? 

BGP was not accounting for any CPU. 

1B) Is the line card where the provider circuit for which you are turning up BGP showing high load? In my experience, anywhere about 45 or 50 will cause issues. 

All the line cards were showing low utilization. 

2) Do you have enough space in your FIB to account for all of the 'best' routes that are being installed? I think the MLX has a limit of 512,000 FIB entries. If you're trying to install more routes then that, perhaps what you are seeing is the result of the popping out of the old entities -- I've seen similar performance degradation under this circumstance. 

We are essentially upgrading from the MLX32 to the MLXE8 for this reason. The new router (MLXE8) supports 1M routes. The current router (MLX32) supports 512K, but we're not encountering this issue. 

I should point out that we're using the MLXE8 in a stick configuration. The Layer 3 config moves over to the MLXE8 and the MLX32 remains online with Layer 2 config only. 

On Tue, Mar 10, 2015 at 12:05 PM, Jake Mertel < jake at nobistech.net > wrote: 

Couple of thoughts off the top of my head: 

1) When the issue is occurring, what are you seeing in 'show tasks' and 'show cpu lp'? 
1A) Is the BGP task what is taking up your MP CPU in 'show tasks'? 
1B) Is the line card where the provider circuit for which you are turning up BGP showing high load? In my experience, anywhere about 45 or 50 will cause issues. 

2) Do you have enough space in your FIB to account for all of the 'best' routes that are being installed? I think the MLX has a limit of 512,000 FIB entries. If you're trying to install more routes then that, perhaps what you are seeing is the result of the popping out of the old entities -- I've seen similar performance degradation under this circumstance. 

-- 
Regards, 

Jake Mertel 
Nobis Technology Group, LLC 

Web: http://www.nobistech.net 
Phone: 1-480-212-1710 
Mail: 5350 East High Street, Suite 300, Phoenix, AZ 85054 

On Tue, Mar 10, 2015 at 8:30 AM, Chris O'Brien < chris.obrien.1021 at gmail.com > wrote: 

Hi, 

I was looking for some feedback on a strange issue. We are performing a router upgrade. On the new gear, when certain BGP peers are established, the router becomes unresponsive via telnet and the network encounters heavy packet loss/latency. CPU shows idle and the log just shows interfaces flapping. Serial access is OK. When you turn these BGP peers down, the issue goes away. When you turn them back up, the issue returns. It is very reproducible. 

Configs are the same. 

Here is our current setup-- 

SL M1: NI-MLX-32_MR Management Module Active (Serial #:, Part #: 35639-300C): 
Boot : Version 5.3.0T165 Copyright (c) 1996-2009 Brocade Communications Systems, Inc. 
Compiled on Nov 16 2011 at 10:05:30 labeled as xmprm05300 
(517880 bytes) from boot flash 
Monitor : Version 5.3.0T165 Copyright (c) 1996-2009 Brocade Communications Systems, Inc. 
Compiled on Nov 16 2011 at 10:04:52 labeled as xmb05300 
(524496 bytes) from code flash 
IronWare : Version 5.3.0aT163 Copyright (c) 1996-2009 Brocade Communications Systems, Inc. 
Compiled on Apr 6 2012 at 10:27:24 labeled as xmr05300a 
(8063716 bytes) from Primary 
Board ID : 00 MBRIDGE32 Revision : 35 
916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus 
512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3) 
2048 MB DRAM INSTALLED 
2048 MB DRAM ADDRESSABLE 

And, we are migrating to-- 

SL M1: BR-MLX-MR2-X Management Module Active (Serial #: , Part #: 60-1002375-06): 
Boot : Version 5.6.0T165 Copyright (c) 1996-2013 Brocade Communications Systems, Inc. 
Compiled on Sep 20 2013 at 16:42:38 labeled as xmprm05600 
(516258 bytes) from boot flash 
Monitor : Version 5.6.0T165 Copyright (c) 1996-2013 Brocade Communications Systems, Inc. 
Compiled on Sep 20 2013 at 16:41:56 labeled as xmb05600 
(534157 bytes) from code flash 
IronWare : Version 5.6.0dT163 Copyright (c) 1996-2013 Brocade Communications Systems, Inc. 
Compiled on Jul 24 2014 at 22:51:20 labeled as xmr05600d 
(9442445 bytes) from Primary 
Board ID : 00 MBRIDGE Revision : 37 
1666 MHz Power PC processor 7448 (version 8004/0202) 166 MHz bus 
512 KB Boot Flash (MX29LV040C), 128 MB Code Flash (MT28F256J3) 
4096 MB DRAM INSTALLED 
4096 MB DRAM ADDRESSABLE 

_______________________________________________ 
foundry-nsp mailing list 
foundry-nsp at puck.nether.net 
http://puck.nether.net/mailman/listinfo/foundry-nsp 

_______________________________________________
foundry-nsp mailing list
foundry-nsp at puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp