[f-nsp] BigIron 4k with JetCore
Brendan Mannella
bmannella at teraswitch.com
Mon Jul 14 10:36:22 EDT 2008
Here is a copy and paste of the "sh cam-part det". Does it look exhausted?
==== SLOT 1 CAM PARTITION ====
IGC: 0 (0x00)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 2047 (0x007ff)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 2021 (0x007e5)
L3 4096 - 32767 (0x01000 - 0x07fff), free 28049 (0x06d91)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 8059 (0x01f7b)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
IGC: 1 (0x01)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 2047 (0x007ff)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 2048 (0x00800)
L3 4096 - 32767 (0x01000 - 0x07fff), free 28552 (0x06f88)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 8070 (0x01f86)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
==== SLOT 2 CAM PARTITION ====
IGC: 4 (0x04)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 2047 (0x007ff)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 2048 (0x00800)
L3 4096 - 32767 (0x01000 - 0x07fff), free 28552 (0x06f88)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 8070 (0x01f86)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
IGC: 5 (0x05)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 2047 (0x007ff)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 2048 (0x00800)
L3 4096 - 32767 (0x01000 - 0x07fff), free 28552 (0x06f88)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 8070 (0x01f86)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
==== SLOT 3 CAM PARTITION ====
IGC: 8 (0x08)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 1964 (0x007ac)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 1118 (0x0045e)
L3 4096 - 32767 (0x01000 - 0x07fff), free 13172 (0x03374)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 7967 (0x01f1f)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
IGC: 9 (0x09)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 2043 (0x007fb)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 1833 (0x00729)
L3 4096 - 32767 (0x01000 - 0x07fff), free 21725 (0x054dd)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 8054 (0x01f76)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
IGC: 10 (0x0a)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 2047 (0x007ff)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 2048 (0x00800)
L3 4096 - 32767 (0x01000 - 0x07fff), free 28552 (0x06f88)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 8070 (0x01f86)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
IGC: 11 (0x0b)
Number of CAM devices per IGC: 1
Number of hw entries per CAM: 0x08000
Total size of CAM = 2Mbits
complete CAM index range per IGC:
(sw) 1 - 49151 (1 - 0x0bfff), total entries: 49151 (0x0bfff)
(hw) 0 - 32767 (0 - 0x07fff), total entries: 32768 (0x08000)
Percentage of CAM hardware entries for each partition:
Layer3 = 16384 (1Mbits) (50%)
Level3 = 1023 (0.062438Mbits) (3.121948%)
Level2 = 1024 (0.0625Mbits) (3.125%)
Layer2 = 8192 (0.5Mbits) (25%)
Layer4 = 8192 (0.5Mbits) (25%)
Layer 3 sw index range:
L3 L3 1 - 2047 (0x00001 - 0x007ff), free 2047 (0x007ff)
L3 L2 2048 - 4095 (0x00800 - 0x00fff), free 2048 (0x00800)
L3 4096 - 32767 (0x01000 - 0x07fff), free 28552 (0x06f88)
layer 3 hw index range (inversely mapped):
16383 - 0 (0x03fff - 0x00000)
L2 index range:
(sw) 32768 - 40959 (0x08000 - 0x09fff), free 8070 (0x01f86)
(hw) 16384 - 24575 (0x04000 - 0x05fff)
L4 pool 3 index range:
(sw) 40960 - 43519 (0x0a000 - 0x0a9ff), free 2560 (0x00a00)
(hw) 24576 - 27135 (0x06000 - 0x069ff)
L4 pool 2 index range:
(sw) 43520 - 46079 (0x0aa00 - 0x0b3ff), free 2560 (0x00a00)
(hw) 27136 - 29695 (0x06a00 - 0x073ff)
L4 pool 1 index range:
(sw) 46080 - 48639 (0x0b400 - 0x0bdff), free 2560 (0x00a00)
(hw) 29696 - 32255 (0x07400 - 0x07dff)
L4 pool 0 index range:
(sw) 48640 - 49151 (0x0be00 - 0x0bfff), free 504 (0x001f8)
(hw) 32256 - 32767 (0x07e00 - 0x07fff)
On 7/14/08 6:16 AM, "Jeroen Wunnink" <jeroen at easyhosting.nl> wrote:
> Might want to take a look at the free cam entries and check if
> they're not very low (sh cam-part det)
> We've had some issues with the L3 cam getting very low and cpu
> spiking. Re-assigning some L2 cam space to L3 cam space resolved
> this. ('cam-partition l2 7 l3 68 l4 25' for example is what worked
> out good for us)
>
> Also look if you're not running low on the max. amount of ip routes
> or hitting the max. amount of ip cache settings ('sh ip route' and
> 'sh ip cache'), the default is set to something like 140000, setting
> those 2 to 400000 helps a lot. ('system-max ip-cache 400000' and
> 'system-max ip-route 400000'), this can be another reason for high CPU spikes.
>
> Take note that every 4 gigabit ports on a Bigiron Jetcore modules
> share the same CAM space, so if you're running low on CAM on a
> certain set of ports it may indeed be a good idea to share the load
> between more port sets.
>
> From the foundry CAM whitepaper:
> JetCore ASIC:
>
> The JetCore products are composed of the JetCore "building blocks,"
> the IGC and the IPC. Each IGC supports four Gigabit Ethernet ports,
> and each IPC supports 24 Fast Ethernet ports plus one Gig port. As
> an illustration of how the IGC and IPC are used, the FWS4802 consists
> of two IPCs linked together. An 8-port Gigabit module has two IGCs,
> and the FI2404 module has one IGC and one IPC. Each IGC or IPC has
> its own CAM space.
>
> Each IPC or IGC has 1Mbit of CAM for FastIron modules or 2Mbit for
> BigIron modules. Therefore, a J-BxG has 4Mbit total, a J-FI48E has
> 2Mbit, and a J-B16GC has 8Mbit.
>
>
> At 00:18 14-7-2008, you wrote:
>> Hello,
>>
>> I am running into a issue with my BI4k CPU spiking and hold timers
>> expiring on my BGP sessions with the border routers. My network
>> consists of 2 Juniper M7i's acting as border routers, each with a
>> transit attached and receiving full routes. Both M7is are connected
>> to the BI4k and run IBGP. Then i have a 16 port copper gige blade,
>> also JetCore. All of my top of rack switches connect to the 16 port
>> card. The top of rack switches are just L2 with tagged uplink ports.
>> Now each customer gets there own VLAN and /29 or larger block. So i
>> have a bunch of ve's so the BI4k acts as the default gateway for the
>> customers ip blocks. My normal traffic aggregate traffic is about 50
>> meg or so max, but last night i have a 50 meg traffic spike and
>> exactly at that time i lost my session between the M7i and the BI4k.
>>
>> I have no idea what limitation i am hitting, but this box should be
>> good for much more than this.
>>
>> The local Foundry SE says i should spread my uplink ports from my L2
>> switches across the 16 port blade. He seems to think because i have
>> all 5 switches plugged right next to each other that i am exhausting
>> CAM resources of the card. The card has 4 ground of 4 ports, he
>> wants me to plug one uplink per group of 4. But i cant believe thats
>> all this card can take.
>>
>> Can anyone shed any light on this?
>>
>> Thanks,
>>
>> Brendan
>> _______________________________________________
>> foundry-nsp mailing list
>> foundry-nsp at puck.nether.net
>> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
>
>
> Met vriendelijke groet,
>
> Jeroen Wunnink,
> EasyHosting B.V. Systeembeheerder
> systeembeheer at easyhosting.nl
>
> telefoon:+31 (035) 6285455 Postbus 48
> fax: +31 (035) 6838242 3755 ZG Eemnes
>
> http://www.easyhosting.nl
> http://www.easycolocate.nl
>
>
More information about the foundry-nsp
mailing list