[f-nsp] CES FIB capacity revisited

[David Unfried]

The numbers in the discussion you reference is for the CER. You are asking
about and have CES deployed.

While the CES/CER are the same software platform, and probably more alike in
hardware than anyone at Brocade will admit, they are different products
marketed and sold with different feature sets and capabilities. The CER is
primarily a router that does some switching. The CES is a layer 2 switch
that also does some routing.
I don't know the numbers, but the CES total routes in FIB is a significantly
lower number than the CER. 
You might be able to order a software key "unlocking" a higher number.

-----Original Message-----
From: foundry-nsp [mailto:foundry-nsp-bounces at puck.nether.net] On Behalf Of
Tamihiro Yuzawa
Sent: Tuesday, August 20, 2013 7:18 PM
To: foundry-nsp at puck.nether.net
Subject: [f-nsp] CES FIB capacity revisited

Helllo experts, especially Mr. Greg Hankins!

A datacenter guy in Japan newly joining the list, please correct me in case
whatever I'm saying is even slightly inappropriate here.

I'm following this informative thread:
https://puck.nether.net/pipermail/foundry-nsp/2012-October/003719.html

because there is a particular VRRP pair of CES2048CX boxes on our site that
have issues with making FIB entries.

> Hi Rob, the CER routers use SRAM for the IPv4/v6 FIB instead of CAM, 
> so for the CES/CER/CER-RT platforms the scalability is dynamic for all 
> IP routes in the FIB (vs the MLX CAM architecture where we have to 
> choose fixed partition sizes which can support a maximum number).

Thank you Greg for the information.
We had a Brocade Japan field engineer visiting our office yesterday, and he
didn't realize CES would use SRAM for FIB instead of CAM.

> Scalability depends on how the memory is used by IPv4 and IPv6 routes.
> There is no easy formula to calculate the utilization, so we have 
> tested certain combinations which are officially supported maximums.  
> While a different number of routes might work, it will not be a 
> supported or tested combination.

Very understandable.

> "Internet route mix" is used to indicate that we are using a mix of 
> prefix lengths.  No compression is being used.

Understood.

And here is our situation.
Since the 3rd of August, we have started seeing a "Warning: IPv4 Network
Route ADD: CAM entry creation FAILED" message every time we try to add a
static IPv4 route, and a "show ip route detail" command for the prefix
returns the following:

>       U_flags   Entry_flags  Age   Cam:Index               HW_Path_count
>       0000e000               0     INVALID CAM             1
> 
>         CAM Entry Flag: 00000000H
>         No cam index

As such, the routers fail to forward packets destined to this prefix towards
the corrrect NH. They just send packets to the deafault NH. 
(While it seems misleading they keep saying CAM when it's actually RAM, I
don't mean to argue with that now :P)

However, the number of routes they have is only <1200 (IPv4) and <500
(IPv6), which is far below the "officially supported maximum" that Greg
kindly shared in his previous email.

Here is a routing protocol breakdown of IPv4 routes in case it matters:
D:   21
O:  860
O2:  74
S:  213

I should also mention that they are mostly /27 or longer prefix lengths.

pref   route
len    count
=====  =====
0:        2
16:       2
17:       2
18:       2
19:       0
20:       8
21:       2
22:       6
23:      80
24:      22
25:      14
26:      45
27:     134
28:     521
29:      85
30:      95
31:      24
32:     124

What's more is that except the two default (/0) and a few others, they are
all subnets of either of two /21 prefixes. 
So I would say our routes are a far from evenly distributed mix of prefixes
which is presumably unlike the "Internet route mix" that Greg used for
testing.

I assume if FIB in SRAM is stored in a tree structure, how many routes it
can hold pretty much depends on the combination of what kind of prefix mix
will be injected and how it's designed to store them.

And my guess is that in our case, the aforementioned unevenness might be
hitting some scale limitations on FIB in CES, be it SRAM partitions, leaf
split constraints, or something beyond my imagination.

I'd appreciate insights from experts, hints, or suggestions for further
investigations because this is becoming a serous issue since it's happening
on our production PE routers. Thank you, -tami
_______________________________________________
foundry-nsp mailing list
foundry-nsp at puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp