[f-nsp] "Low pool memory, Pool(2-Global) below 5%" on Brocade MLX box

[Chris Hellkvist]

Dear all,

recently we connected another upstream to a Brocade MLX box.

We noticed that the MLX was not learning all routers from this BGP
peer both on the IPv4 and IPv6 session. Looking for the root cause we
found the following entries in the log of the box:
Mar  7 11:46:59:W:System: Low pool memory, Pool(2-Global) below 5%,
total memory(1073741824), available memory(53534720) on MP
Mar  7 11:46:58:W:System: Low physical memory, Pool(2-Global) below
5%, total memory(1073741824), available memory(53235712) on MP

I'm a bit worried because the system has not that many BGP sessions
active (around 8 while 4 peers send a full table that is reduces to
around 390k routes in the RIB/FIB using prefix lists).

After removing the BGP peers the free memory went to arund 7%, however
prior to the configuration we have never had any warning regarding the
 usual 10% threshold).

Looking at the memory histogram i can see that most memory is consumed
by the following processes:
bgp                          1221            106342047
bgp_io                       3925            473596101

Any idea how to reduce the memory footprint of bgp_io or to free up
more space by adjusting some system-max variables? Or any other
options than replacing the box with some new gear (luckily this will
happen  end of 2016).

Thanks,
Chris

For the recordds, the modules we have in place and a sh ve of the system:
SSH at CRBR13-F#sh module
Module                                                      Status
                  Ports    Starting MAC
M1 (left ):NI-MLX-MR Management Module                      Active
M2 (right):NI-MLX-MR Management Module                      Standby(Ready State)
F1: NI-X-SF  Switch Fabric Module                           Active
F2: NI-X-SF  Switch Fabric Module                           Active
F3: NI-X-SF  Switch Fabric Module                           Active
S1: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP                4        001b.ed17.d000
S2: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP                4        001b.ed17.d030
S3: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP                4        001b.ed17.d060
S4: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP                4        001b.ed17.d090
S5: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP                4        001b.ed17.d0c0

SSH at CRBR13-F#sh ve
System Mode: MLX
Chassis: NetIron 8-slot (Serial #: A60532F06T,  Part #: 35549-000C)
NI-X-SF  Switch Fabric Module 1 (Serial #: SA38070273,  Part #: 35523-300C)
FE 1: Type fe200,  Version 2
FE 3: Type fe200,  Version 2
Switch Fabric Module 1 Up Time is 286 days 23 hours 8 minutes 55 seconds
NI-X-SF  Switch Fabric Module 2 (Serial #: SA13070172,  Part #: 35523-300C)
FE 1: Type fe200,  Version 2
FE 3: Type fe200,  Version 2
Switch Fabric Module 2 Up Time is 286 days 23 hours 8 minutes 55 seconds
NI-X-SF  Switch Fabric Module 3 (Serial #: SA13080287,  Part #: 35523-301A)
FE 1: Type fe200,  Version 2
FE 3: Type fe200,  Version 2
Switch Fabric Module 3 Up Time is 286 days 23 hours 8 minutes 55 seconds
==========================================================================
SL M1: NI-MLX-MR Management Module Active (Serial #: S10526F06N, Part
#:    35524-103J):
Boot     : Version 5.6.0T165 Copyright (c) 1996-2014 Brocade
Communications Systems, Inc.
Compiled on Sep 20 2013 at 16:42:38 labeled as xmprm05600
 (516258 bytes) from boot flash
Monitor  : Version 5.7.0T165 Copyright (c) 1996-2014 Brocade
Communications Systems, Inc.
Compiled on May  5 2014 at 10:41:24 labeled as xmb05700
 (537647 bytes) from code flash
IronWare : Version 5.7.0cT163 Copyright (c) 1996-2014 Brocade
Communications Systems, Inc.
Compiled on Feb 25 2015 at 20:44:00 labeled as xmr05700c
 (9651771 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 37
916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus
512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
1024 MB DRAM INSTALLED
1024 MB DRAM ADDRESSABLE
Active Management uptime is 286 days 23 hours 8 minutes 55 seconds