[f-nsp] "Low pool memory, Pool(2-Global) below 5%" on Brocade MLX box
Eldon Koyle
ekoyle+puck.nether.net at gmail.com
Tue Mar 8 13:17:43 EST 2016
I thought 5.6 was the last supported release for the 1st gen hardware
(-MR management module, 10Gx4 cards, etc.), although I'm not seeing
anything about the MR module so maybe it is still supported.
>From 5.7 release notes:
Note: Gen 1 Module SKUs are not supported beyond R05.6:00
• NI-MLX-1Gx20-GC
• NI-MLX-1Gx20-SFP
• NI-XMR-1Gx20-GC
• NI-XMR-1Gx20-SFP
• NI-MLX-10Gx2
• NI-MLX-10Gx4
I don't know if that is part of your problem, but is there a feature
you needed that wasn't available in 05600g? If not, you might
consider downgrading to see if there is a difference there.
It is possible that there is a memory leak somewhere in 5.7 or
something like that.
The MR modules only had 1 GB of ram, the MR2 has 4 GB. You could
upgrade to a single MR2 management module in this device (this is what
they use in the MLXe chassis, if you are sticking with brocade for
your replacement).
--
Eldon
On Tue, Mar 8, 2016 at 10:17 AM, Chris Hellkvist
<chris.hellkvist at googlemail.com> wrote:
> Sure:
>
> SSH at CRBR13-F> sh cam-partition brief
>
> CAM partitioning profile: multi-service-2
>
> Slot 1 XPP20SP 0:
> # of CAM device = 3
> Total CAM Size = 524288 entries (36Mbits)
>
> IP: Raw Size 458752, User Size 458752(0 reserved)
> Subpartition 0: Raw Size 2048, User Size 2048, (0 reserved)
> Subpartition 1: Raw Size 411289, User Size 411289, (0 reserved)
> Subpartition 2: Raw Size 37892, User Size 37892, (0 reserved)
> Subpartition 3: Raw Size 5741, User Size 5741, (0 reserved)
> Subpartition 4: Raw Size 1112, User Size 1112, (0 reserved)
>
> IPv6: Raw Size 32768, User Size 16384(0 reserved)
> Subpartition 0: Raw Size 2048, User Size 1024, (0 reserved)
> Subpartition 1: Raw Size 27204, User Size 13602, (0 reserved)
> Subpartition 2: Raw Size 2332, User Size 1166, (0 reserved)
> Subpartition 3: Raw Size 512, User Size 256, (0 reserved)
> Subpartition 4: Raw Size 384, User Size 192, (0 reserved)
>
> SSH at CRBR13-F>sh default values
> sys log buffers:50 mac age time:300 sec telnet sessions:5
>
> ip arp age:10 min bootp relay max hops:4 ip ttl:64 hops
> ip addr per intf:24
>
> when multicast enabled :
> igmp group memb.: 260 sec igmp query: 125 sec
>
> when ospf enabled :
> ospf dead:40 sec ospf hello:10 sec ospf retrans:5 sec
> ospf transit delay:1 sec
>
> when bgp enabled :
> bgp local pref.:100 bgp keep alive:60 sec bgp hold:180 sec
> bgp metric:10 bgp local as:1 bgp cluster id:0
> bgp ext. distance:20 bgp int. distance:200 bgp local distance:200
>
> when IS-IS enabled :
> isis hello interval:10 sec isis hello multiplier:3
> isis port metric:10 isis priority:64
> isis csnp-interval:10 sec isis default-metric:10
> isis distance:115 isis lsp-gen-interval:10 sec
> isis lsp-interval:33 msec isis lsp-refresh-interval:900 sec
> isis max-lsp-lifetime:1200 sec isis maximum-paths:4
> isis retransmit-interval:5 sec isis spf-interval:5 sec
>
> filter change update delay:10 sec
>
> System Parameters Default Maximum Current Actual
> Bootup Revertible
> mac 32768 1048576 32768 32768
> 32768 Yes
> vlan 512 4095 1024 1024
> 1024 No
> spanning-tree 32 128 32 32
> 32 No
> rstp 32 128 32 32
> 32 No
> ip-arp 8192 65536 40000 40000
> 40000 No
> ip-cache 102400 524288 440000 440000
> 440000 Yes
> ip-route 102400 524288 440000 440000
> 440000 Yes
> ip-subnet-port 24 128 24 24
> 24 No
> virtual-interface 255 4095 800 800
> 800 No
> vpls-mac 2048 262144 2048 2048
> 2048 Yes
> vpls-num 512 4096 512 512
> 512 No
> session-limit 8192 40960 8192 8192
> 8192 Yes
> ip-filter-sys 4096 102400 4096 4096
> 4096 No
> mgmt-port-acl-size 20 100 20 20
> 20 No
> l2-acl-table-entries 64 256 64 64
> 64 No
> ipv6-cache 32768 114688 16000 16000
> 16000 Yes
> ipv6-route 32768 114688 16000 16000
> 16000 Yes
> ip-vrf-route 5120 262144 5120 5120
> 5120 Yes
> receive-cam 1024 16384 1024 1024
> 1024 No
> ip-tunnels 256 8192 256 256
> 256 No
> lsp-out-acl-cam 0 16384 0 0 0
> No
> trunk-num 128 256 128 128
> 128 No
> config-file-size 8388608 32505856 8388608 8388608
> 8388608 No
> ifl-cam 8192 81920 8192 8192
> 8192 No
> ip-source-guard-cam 0 131072 0 0 0
> No
> ipv4-mcast-cam 4096 32768 4096 4096
> 4096 No
> ipv6-mcast-cam 1024 8192 1024 1024
> 1024 No
> mcast-vpls-cam 0 2048 0 0 0
> Yes
> ip-vrf 400 400 400 400
> 400 Yes
> ipv6-vrf-route 128 16384 128 128
> 128 Yes
> subnet-broadcast-acl-cam 0 4096 0 0 0
> No
> trunk-num-100g 8 64 8 8 8
> No
> openflow-flow-entries 0 65536 0 0 0
> Yes
> openflow-pvlan-entries 0 2048 0 0 0
> Yes
> openflow-unprotectedvlan 0 4096 0 0 0
> Yes
> ipv6-receive-cam 0 8192 0 0 0
> No
> ecmp-pram-block-size 32 32 32 32
> 32 Yes
> tvf-lag-lb-fid-pool 0 2048 0 0 0
> Yes
> tvf-lag-lb-fid-group 4 16 4 4 4
> Yes
> openflow-group-select-bu 8 64 8 8 8
> Yes
>
> Openflow Slot Parameters slot Default
> Maximum Actual Revertible
> np-openflow-flow-entries : layer2or3 1 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 2 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 3 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 4 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 5 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 6 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 7 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 8 0
> 32768 0 No
>
> np-openflow-flow-entries : layer23ipv4 1 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 2 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 3 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 4 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 5 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 6 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 7 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 8 0
> 32768 0 No
>
> 2016-03-08 18:00 GMT+01:00 Eldon Koyle <ekoyle+puck.nether.net at gmail.com>:
>> Could you also provide the output of 'show default values'?
>>
>> You may need to change your cam-partition profile.
>>
>> --
>> Eldon
>>
>> On Tue, Mar 8, 2016 at 8:13 AM, Chris Hellkvist via foundry-nsp
>> <foundry-nsp at puck.nether.net> wrote:
>>> Dear all,
>>>
>>> recently we connected another upstream to a Brocade MLX box.
>>>
>>> We noticed that the MLX was not learning all routers from this BGP
>>> peer both on the IPv4 and IPv6 session. Looking for the root cause we
>>> found the following entries in the log of the box:
>>> Mar 7 11:46:59:W:System: Low pool memory, Pool(2-Global) below 5%,
>>> total memory(1073741824), available memory(53534720) on MP
>>> Mar 7 11:46:58:W:System: Low physical memory, Pool(2-Global) below
>>> 5%, total memory(1073741824), available memory(53235712) on MP
>>>
>>> I'm a bit worried because the system has not that many BGP sessions
>>> active (around 8 while 4 peers send a full table that is reduces to
>>> around 390k routes in the RIB/FIB using prefix lists).
>>>
>>> After removing the BGP peers the free memory went to arund 7%, however
>>> prior to the configuration we have never had any warning regarding the
>>> usual 10% threshold).
>>>
>>> Looking at the memory histogram i can see that most memory is consumed
>>> by the following processes:
>>> bgp 1221 106342047
>>> bgp_io 3925 473596101
>>>
>>> Any idea how to reduce the memory footprint of bgp_io or to free up
>>> more space by adjusting some system-max variables? Or any other
>>> options than replacing the box with some new gear (luckily this will
>>> happen end of 2016).
>>>
>>> Thanks,
>>> Chris
>>>
>>> For the recordds, the modules we have in place and a sh ve of the system:
>>> SSH at CRBR13-F#sh module
>>> Module Status
>>> Ports Starting MAC
>>> M1 (left ):NI-MLX-MR Management Module Active
>>> M2 (right):NI-MLX-MR Management Module Standby(Ready State)
>>> F1: NI-X-SF Switch Fabric Module Active
>>> F2: NI-X-SF Switch Fabric Module Active
>>> F3: NI-X-SF Switch Fabric Module Active
>>> S1: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d000
>>> S2: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d030
>>> S3: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d060
>>> S4: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d090
>>> S5: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d0c0
>>>
>>> SSH at CRBR13-F#sh ve
>>> System Mode: MLX
>>> Chassis: NetIron 8-slot (Serial #: A60532F06T, Part #: 35549-000C)
>>> NI-X-SF Switch Fabric Module 1 (Serial #: SA38070273, Part #: 35523-300C)
>>> FE 1: Type fe200, Version 2
>>> FE 3: Type fe200, Version 2
>>> Switch Fabric Module 1 Up Time is 286 days 23 hours 8 minutes 55 seconds
>>> NI-X-SF Switch Fabric Module 2 (Serial #: SA13070172, Part #: 35523-300C)
>>> FE 1: Type fe200, Version 2
>>> FE 3: Type fe200, Version 2
>>> Switch Fabric Module 2 Up Time is 286 days 23 hours 8 minutes 55 seconds
>>> NI-X-SF Switch Fabric Module 3 (Serial #: SA13080287, Part #: 35523-301A)
>>> FE 1: Type fe200, Version 2
>>> FE 3: Type fe200, Version 2
>>> Switch Fabric Module 3 Up Time is 286 days 23 hours 8 minutes 55 seconds
>>> ==========================================================================
>>> SL M1: NI-MLX-MR Management Module Active (Serial #: S10526F06N, Part
>>> #: 35524-103J):
>>> Boot : Version 5.6.0T165 Copyright (c) 1996-2014 Brocade
>>> Communications Systems, Inc.
>>> Compiled on Sep 20 2013 at 16:42:38 labeled as xmprm05600
>>> (516258 bytes) from boot flash
>>> Monitor : Version 5.7.0T165 Copyright (c) 1996-2014 Brocade
>>> Communications Systems, Inc.
>>> Compiled on May 5 2014 at 10:41:24 labeled as xmb05700
>>> (537647 bytes) from code flash
>>> IronWare : Version 5.7.0cT163 Copyright (c) 1996-2014 Brocade
>>> Communications Systems, Inc.
>>> Compiled on Feb 25 2015 at 20:44:00 labeled as xmr05700c
>>> (9651771 bytes) from Primary
>>> Board ID : 00 MBRIDGE Revision : 37
>>> 916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus
>>> 512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
>>> 1024 MB DRAM INSTALLED
>>> 1024 MB DRAM ADDRESSABLE
>>> Active Management uptime is 286 days 23 hours 8 minutes 55 seconds
>>> _______________________________________________
>>> foundry-nsp mailing list
>>> foundry-nsp at puck.nether.net
>>> http://puck.nether.net/mailman/listinfo/foundry-nsp
More information about the foundry-nsp
mailing list