[f-nsp] CES FIB capacity revisited

Tamihiro Yuzawa tamil at tox.mine.nu
Thu Aug 22 00:13:07 EDT 2013 

Hi David, thank you for the clarification. 
I should have referred to their datasheets as for the specific FIB size of the CES.
http://www.brocade.com/downloads/documents/data_sheets/product_data_sheets/netiron-ces-2000-ds.pdf

Yes, compared to CER, FIB in CES is very small.
Especially now the CER (without RT) has as much capacity as the former CER-RT. 

But still, they say the CES series can store... 

- Up to 32,000 IPv4 unicast routes in FIB
- Up to 8000 IPv6 unicast routes in FIB

Please take a look at another email that I posted on this thread about an hour ago.
The problem has gone after a reboot :)

Let's wait and see what their TAC has to say. 
Best regards, -tami


dunfried> The numbers in the discussion you reference is for the CER. You are asking
dunfried> about and have CES deployed.
dunfried> 
dunfried> While the CES/CER are the same software platform, and probably more alike in
dunfried> hardware than anyone at Brocade will admit, they are different products
dunfried> marketed and sold with different feature sets and capabilities. The CER is
dunfried> primarily a router that does some switching. The CES is a layer 2 switch
dunfried> that also does some routing.
dunfried> I don't know the numbers, but the CES total routes in FIB is a significantly
dunfried> lower number than the CER. 
dunfried> You might be able to order a software key "unlocking" a higher number.
dunfried> 
dunfried> -----Original Message-----
dunfried> From: foundry-nsp [mailto:foundry-nsp-bounces at puck.nether.net] On Behalf Of
dunfried> Tamihiro Yuzawa
dunfried> Sent: Tuesday, August 20, 2013 7:18 PM
dunfried> To: foundry-nsp at puck.nether.net
dunfried> Subject: [f-nsp] CES FIB capacity revisited
dunfried> 
dunfried> Helllo experts, especially Mr. Greg Hankins!
dunfried> 
dunfried> A datacenter guy in Japan newly joining the list, please correct me in case
dunfried> whatever I'm saying is even slightly inappropriate here.
dunfried> 
dunfried> I'm following this informative thread:
dunfried> https://puck.nether.net/pipermail/foundry-nsp/2012-October/003719.html
dunfried> 
dunfried> because there is a particular VRRP pair of CES2048CX boxes on our site that
dunfried> have issues with making FIB entries.
dunfried> 
dunfried> > Hi Rob, the CER routers use SRAM for the IPv4/v6 FIB instead of CAM, 
dunfried> > so for the CES/CER/CER-RT platforms the scalability is dynamic for all 
dunfried> > IP routes in the FIB (vs the MLX CAM architecture where we have to 
dunfried> > choose fixed partition sizes which can support a maximum number).
dunfried> 
dunfried> Thank you Greg for the information.
dunfried> We had a Brocade Japan field engineer visiting our office yesterday, and he
dunfried> didn't realize CES would use SRAM for FIB instead of CAM.
dunfried> 
dunfried> > Scalability depends on how the memory is used by IPv4 and IPv6 routes.
dunfried> > There is no easy formula to calculate the utilization, so we have 
dunfried> > tested certain combinations which are officially supported maximums.  
dunfried> > While a different number of routes might work, it will not be a 
dunfried> > supported or tested combination.
dunfried> 
dunfried> Very understandable.
dunfried> 
dunfried> > "Internet route mix" is used to indicate that we are using a mix of 
dunfried> > prefix lengths.  No compression is being used.
dunfried> 
dunfried> Understood.
dunfried> 
dunfried> And here is our situation.
dunfried> Since the 3rd of August, we have started seeing a "Warning: IPv4 Network
dunfried> Route ADD: CAM entry creation FAILED" message every time we try to add a
dunfried> static IPv4 route, and a "show ip route detail" command for the prefix
dunfried> returns the following:
dunfried> 
dunfried> >       U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
dunfried> >       0000e000               0     INVALID CAM             1
dunfried> > 
dunfried> >         CAM Entry Flag: 00000000H
dunfried> >         No cam index
dunfried> 
dunfried> As such, the routers fail to forward packets destined to this prefix towards
dunfried> the corrrect NH. They just send packets to the deafault NH. 
dunfried> (While it seems misleading they keep saying CAM when it's actually RAM, I
dunfried> don't mean to argue with that now :P)
dunfried> 
dunfried> However, the number of routes they have is only <1200 (IPv4) and <500
dunfried> (IPv6), which is far below the "officially supported maximum" that Greg
dunfried> kindly shared in his previous email.
dunfried> 
dunfried> Here is a routing protocol breakdown of IPv4 routes in case it matters:
dunfried> D:   21
dunfried> O:  860
dunfried> O2:  74
dunfried> S:  213
dunfried> 
dunfried> I should also mention that they are mostly /27 or longer prefix lengths.
dunfried> 
dunfried> pref   route
dunfried> len    count
dunfried> =====  =====
dunfried> 0:        2
dunfried> 16:       2
dunfried> 17:       2
dunfried> 18:       2
dunfried> 19:       0
dunfried> 20:       8
dunfried> 21:       2
dunfried> 22:       6
dunfried> 23:      80
dunfried> 24:      22
dunfried> 25:      14
dunfried> 26:      45
dunfried> 27:     134
dunfried> 28:     521
dunfried> 29:      85
dunfried> 30:      95
dunfried> 31:      24
dunfried> 32:     124
dunfried> 
dunfried> What's more is that except the two default (/0) and a few others, they are
dunfried> all subnets of either of two /21 prefixes. 
dunfried> So I would say our routes are a far from evenly distributed mix of prefixes
dunfried> which is presumably unlike the "Internet route mix" that Greg used for
dunfried> testing.
dunfried> 
dunfried> I assume if FIB in SRAM is stored in a tree structure, how many routes it
dunfried> can hold pretty much depends on the combination of what kind of prefix mix
dunfried> will be injected and how it's designed to store them.
dunfried> 
dunfried> And my guess is that in our case, the aforementioned unevenness might be
dunfried> hitting some scale limitations on FIB in CES, be it SRAM partitions, leaf
dunfried> split constraints, or something beyond my imagination.
dunfried> 
dunfried> I'd appreciate insights from experts, hints, or suggestions for further
dunfried> investigations because this is becoming a serous issue since it's happening
dunfried> on our production PE routers. Thank you, -tami
dunfried> _______________________________________________
dunfried> foundry-nsp mailing list
dunfried> foundry-nsp at puck.nether.net
dunfried> http://puck.nether.net/mailman/listinfo/foundry-nsp
dunfried> 
dunfried>

More information about the foundry-nsp mailing list