[f-nsp] CES FIB capacity revisited
Tamihiro Yuzawa
tamil at tox.mine.nu
Wed Aug 21 23:17:46 EDT 2013
Greg-san, thanks so much for your suggestion!
I have some interesting updates.
On Monday during the late hours, we evacuated some of the downlink customers to another PE pair.
Even then, with only 730 IPv4 and 410 IPv6 routes, installing a single new /32 static route still failed.
Today, earlier in the morning we rebooted these boxes, one by one after driving off the incoming traffic to the other, and brought the service traffic back into the boxes.
And now, there's no problem with a new static route getting into FIB :)
U_flags Entry_flags Age Cam:Index HW_Path_count
0000e000 0 0002ce84 1
CAM Entry Flag: 00000001H
PPCR : 1:1 CIDX: 0002ce84 (IP_NETWORK: 0x2ce84)
#sh ip route sum
IP Routing Table - 731 entries
15 connected, 158 static, 0 RIP, 558 OSPF, 0 BGP, 0 ISIS
Number of prefixes:
/0: 1 /16: 1 /17: 1 /18: 1 /20: 4 /21: 1 /22: 3 /23: 40 /24: 19 /25: 7 /26: 23 /27: 68 /28: 275 /29: 74 /30: 92 /31: 24 /32: 97
Nexthop Table Entry - 161 entries
#sh ipv6 route sum
IPv6 Routing Table - 410 entries:
15 connected, 5 static, 0 RIP, 390 OSPF, 0 BGP, 0 ISIS
Number of prefixes:
/0:1 /32:1 /34:1 /64:386 /128:21
So what was it.. I mean before reboots?
Is it fair to suspect that memory fragmentation may have got too bad, compaction work wasn't running as it should, and a reboot was the only way to get some free space back into RAM?
Is there any way to get total SRAM size for FIB vs used amount via CLI?
We have already spoken to our VAR and requested that this issue be escalated to Brocade TAC promptly.
I believe they will reach out for Brocade US engineers for further investigation.
Greg-san, if you could follow up in any way, I'd much appreciate it.
We have deployed a bunch of CES pairs that took over FESX in our datacenters.
Therefore identifying the cause of this issue is very important. -tami
ghankins> Yuzawa-san, this really sounds like a technical issue that needs to be
ghankins> investigated by Brocade TAC and not by your VAR. The number of routes,
ghankins> types of routes and the prefix distribution is so small that you should
ghankins> not be seeing any FIB capacity issues on the CES.
ghankins>
ghankins> Kind regards,
ghankins> Greg
ghankins> (works for Brocade)
ghankins>
ghankins> --
ghankins> Greg Hankins <ghankins at mindspring.com>
ghankins>
ghankins> -----Original Message-----
ghankins> Date: Wed, 21 Aug 2013 11:18:20 +0900 (JST)
ghankins> From: Tamihiro Yuzawa <tamil at tox.mine.nu>
ghankins> To: foundry-nsp at puck.nether.net
ghankins> Subject: [f-nsp] CES FIB capacity revisited
ghankins>
ghankins> Helllo experts, especially Mr. Greg Hankins!
ghankins>
ghankins> A datacenter guy in Japan newly joining the list, please correct me in case whatever I'm saying is even slightly inappropriate here.
ghankins>
ghankins> I'm following this informative thread:
ghankins> https://puck.nether.net/pipermail/foundry-nsp/2012-October/003719.html
ghankins>
ghankins> because there is a particular VRRP pair of CES2048CX boxes on our site that have issues with making FIB entries.
ghankins>
ghankins> > Hi Rob, the CER routers use SRAM for the IPv4/v6 FIB instead of CAM, so
ghankins> > for the CES/CER/CER-RT platforms the scalability is dynamic for all IP
ghankins> > routes in the FIB (vs the MLX CAM architecture where we have to choose
ghankins> > fixed partition sizes which can support a maximum number).
ghankins>
ghankins> Thank you Greg for the information.
ghankins> We had a Brocade Japan field engineer visiting our office yesterday, and he didn't realize CES would use SRAM for FIB instead of CAM.
ghankins>
ghankins> > Scalability depends on how the memory is used by IPv4 and IPv6 routes.
ghankins> > There is no easy formula to calculate the utilization, so we have tested
ghankins> > certain combinations which are officially supported maximums. While a
ghankins> > different number of routes might work, it will not be a supported or
ghankins> > tested combination.
ghankins>
ghankins> Very understandable.
ghankins>
ghankins> > "Internet route mix" is used to indicate that we are using a mix of prefix
ghankins> > lengths. No compression is being used.
ghankins>
ghankins> Understood.
ghankins>
ghankins> And here is our situation.
ghankins> Since the 3rd of August, we have started seeing a "Warning: IPv4 Network Route ADD: CAM entry creation FAILED" message every time we try to add a static IPv4 route, and a "show ip route detail" command for the prefix returns the following:
ghankins>
ghankins> > U_flags Entry_flags Age Cam:Index HW_Path_count
ghankins> > 0000e000 0 INVALID CAM 1
ghankins> >
ghankins> > CAM Entry Flag: 00000000H
ghankins> > No cam index
ghankins>
ghankins> As such, the routers fail to forward packets destined to this prefix towards the corrrect NH. They just send packets to the deafault NH.
ghankins> (While it seems misleading they keep saying CAM when it's actually RAM, I don't mean to argue with that now :P)
ghankins>
ghankins> However, the number of routes they have is only <1200 (IPv4) and <500 (IPv6), which is far below the "officially supported maximum" that Greg kindly shared in his previous email.
ghankins>
ghankins> Here is a routing protocol breakdown of IPv4 routes in case it matters:
ghankins> D: 21
ghankins> O: 860
ghankins> O2: 74
ghankins> S: 213
ghankins>
ghankins> I should also mention that they are mostly /27 or longer prefix lengths.
ghankins>
ghankins> pref route
ghankins> len count
ghankins> ===== =====
ghankins> 0: 2
ghankins> 16: 2
ghankins> 17: 2
ghankins> 18: 2
ghankins> 19: 0
ghankins> 20: 8
ghankins> 21: 2
ghankins> 22: 6
ghankins> 23: 80
ghankins> 24: 22
ghankins> 25: 14
ghankins> 26: 45
ghankins> 27: 134
ghankins> 28: 521
ghankins> 29: 85
ghankins> 30: 95
ghankins> 31: 24
ghankins> 32: 124
ghankins>
ghankins> What's more is that except the two default (/0) and a few others, they are all subnets of either of two /21 prefixes.
ghankins> So I would say our routes are a far from evenly distributed mix of prefixes which is presumably unlike the "Internet route mix" that Greg used for testing.
ghankins>
ghankins> I assume if FIB in SRAM is stored in a tree structure, how many routes it can hold pretty much depends on the combination of what kind of prefix mix will be injected and how it's designed to store them.
ghankins>
ghankins> And my guess is that in our case, the aforementioned unevenness might be hitting some scale limitations on FIB in CES, be it SRAM partitions, leaf split constraints, or something beyond my imagination.
ghankins>
ghankins> I'd appreciate insights from experts, hints, or suggestions for further investigations because this is becoming a serous issue since it's happening on our production PE routers. Thank you, -tami
More information about the foundry-nsp
mailing list