[c-nsp] ASR 1002-X FIB scalability (was: Re: ASR-100x intro)
Mack McBride
mack.mcbride at viawest.com
Wed May 29 11:47:21 EDT 2013
You aren't actually looking at the fib information, just CEF (which is not the same).
The CEF table in the RP is not the same as the CEF table on the ESP.
Routes that aren't processed on the QFP get processed on the RP.
Ie. SLOWLY.
The command you are looking for is:
sh plat hard qfp active infra exmem statistics
If the FIB overflows the DRAM, it will start using IRAM.
If the IRAM fills the ESP may become unstable and traffic is offloaded to the RP.
I have not found a command to actually show the CEF information on the ESP.
LR Mack McBride
Network Architect
-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Beck, Andre
Sent: Tuesday, May 28, 2013 3:45 AM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] ASR 1002-X FIB scalability (was: Re: ASR-100x intro)
Re,
On Thu, Apr 11, 2013 at 11:59:03AM +0200, Beck, Andre wrote:
>
> When my 1002-X box with 16GB hits the lab, before anything else, I'll
> set it up to push BGP tables into it using bgpsimple (I also tried
> exabgp but it gets slow as molasses as soon as you try to push some
> 100k prefixes) and see where it goes through the roof.
For starters, here is the ASR 1002-X with 16GB (atm still running 3.7.0 consolidated as it came), flooded with a BGP table of 3.1M prefixes.
I had to change my prefix generator to allow for prefix lengths of up to /27 in order to get there (trivial stateless randomized generator, the pigeonhole principle and all that), so it stresses the 8-8-8-8 mtrie even more badly. Here's some output from the ASR which took that size of table and FIB without a hitch:
1. The RIB side of the game
asr1002-x#sh ip bgp summary
BGP router identifier 192.168.127.12, local AS number 65533 BGP table version is 31438353, main routing table version 31438353
3114278 network entries using 772340944 bytes of memory
3114278 path entries using 348799136 bytes of memory
1018812/1018812 BGP path/bestpath attribute entries using 228213888 bytes of memory
1018812 BGP AS-PATH entries using 153930440 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory BGP using 1503284408 total bytes of memory BGP activity 17033337/13919059 prefixes, 17276315/14162037 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.111.254 4 65522 3114282 7 31438353 0 0 00:42:06 3114278
asr1002-x#sh ip route summary
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory (bytes)
connected 0 3 0 264 864
static 0 0 0 0 0
bgp 65533 360012 2754266 0 274056464 896912064
External: 3114278 Internal: 0 Local: 0
internal 97357 198402256
Total 457369 2754269 0 274056728 1095315184
2. And now for the FIB stuff
asr1002-x#sh ip cef summary
IPv4 CEF is enabled for distributed and running
VRF Default
3114292 prefixes (3114292/0 fwd/non-fwd)
Table id 0x0
Database epoch: 2 (3114292 entries at this epoch)
asr1002-x#sh cef memory summary
CEF has allocated 2136896276 bytes of memory (5919372 bytes overhead)
asr1002-x#sh cef memory
Memory in use/allocated Count
------------------------------------------------------------------
ADJ: GSB : 468/560 ( 83%) [1]
ADJ: NULL adjacency : 436/528 ( 82%) [1]
ADJ: adj sev context : 232/416 ( 55%) [2]
ADJ: adjacency : 3408/3776 ( 90%) [4]
ADJ: mcprp_adj_inject_sb : 16588/16680 ( 99%) [1]
ADJ: request resolve : 4688/4872 ( 96%) [2]
ADJ: sevs : 432/616 ( 70%) [2]
CEF: Brkr Updat : 9704/10440 ( 92%) [8]
CEF: Brkr Update Rec : 312/496 ( 62%) [2]
CEF: Brkr zone : 988/2736 ( 36%) [19]
CEF: Broker : 5684/7432 ( 76%) [19]
CEF: Brokers Array : 180/272 ( 66%) [1]
CEF: EVENT msg chunks : 332/424 ( 78%) [1]
CEF: FIB LC array : 452/544 ( 83%) [1]
CEF: FIB LC stats array : 9860/9952 ( 99%) [1]
CEF: FIB subtree context : 328/880 ( 37%) [6]
CEF: FIBHWIDB : 15444/16640 ( 92%) [13]
CEF: FIBIDB : 6292/7488 ( 84%) [13]
CEF: IPv4 ARP throttle : 2052/2144 ( 95%) [1]
CEF: IPv4 process : 1128/1864 ( 60%) [8]
CEF: OCE get hash callbac : 52/144 ( 36%) [1]
CEF: TABLE msg chunks : 804/896 ( 89%) [1]
CEF: Table rate Monitor S : 180/456 ( 39%) [3]
CEF: arp throttle chunk : 28168/28352 ( 99%) [2]
CEF: cover need : 528/712 ( 74%) [2]
CEF: cover need deagg chu : 376/560 ( 67%) [2]
CEF: dQ elems : 360/544 ( 66%) [2]
CEF: dQ walks : 512/696 ( 73%) [2]
CEF: fib : 824283024/826521936 ( 99%) [24336]
CEF: fib GSB : 23444/23904 ( 98%) [5]
CEF: fib deps : 352/536 ( 65%) [2]
CEF: fib loop sb : 368/552 ( 66%) [2]
CEF: fib_fib_covered chun : 392/576 ( 68%) [2]
CEF: fib_fib_rp_bfd_sb : 2472/2656 ( 93%) [2]
CEF: fib_fib_sr : 4832/5936 ( 81%) [12]
CEF: fib_fib_src_adj_sb : 368/552 ( 66%) [2]
CEF: fib_fib_src_adj_sb_a : 320/504 ( 63%) [2]
CEF: fib_fib_src_interfac : 344/528 ( 65%) [2]
CEF: fib_fib_src_rr_sb : 376/560 ( 67%) [2]
CEF: fib_fib_src_special_ : 344/528 ( 65%) [2]
CEF: fib_head_s : 664/848 ( 78%) [2]
CEF: fib_head_sb chunk : 376/560 ( 67%) [2]
CEF: fib_member : 2352/2720 ( 86%) [4]
CEF: fib_member_sb chunk : 368/552 ( 66%) [2]
CEF: fib_rib_route_update : 5184/5368 ( 96%) [2]
CEF: fib_table_fibswsb_de : 512/696 ( 73%) [2]
CEF: fibhwidb table : 1048580/1048672 ( 99%) [1]
CEF: fibidb table : 1048580/1048672 ( 99%) [1]
CEF: fibswsb ct : 3208/3392 ( 94%) [2]
CEF: fibswsb ctl : 1680/1864 ( 90%) [2]
CEF: hash table : 262152/262336 ( 99%) [2]
CEF: ipv6 feature error c : 5548/5640 ( 98%) [1]
CEF: ipv6 feature error s : 5548/5640 ( 98%) [1]
CEF: ipv6 not cef switche : 1228/1320 ( 93%) [1]
CEF: ipv6 not cef switche : 1228/1320 ( 93%) [1]
CEF: loadinf16 : 928/1112 ( 83%) [2]
CEF: loadinfo2 : 432/616 ( 70%) [2]
CEF: mpls long path exts : 488/672 ( 72%) [2]
CEF: mpls path exts : 368/552 ( 66%) [2]
CEF: nh entry context : 432/616 ( 70%) [2]
CEF: nh entry params : 432/616 ( 70%) [2]
CEF: non_ip entry context : 488/672 ( 72%) [2]
CEF: pathl : 12840/13576 ( 94%) [8]
CEF: pathl ifs : 2880/3432 ( 83%) [6]
CEF: pathl its : 648/832 ( 77%) [2]
CEF: pathloutputchain : 792/976 ( 81%) [2]
CEF: paths : 13352/14088 ( 94%) [8]
CEF: plist dq it : 488/672 ( 72%) [2]
CEF: prefix query msg chu : 2380/2472 ( 96%) [1]
CEF: subtree context : 2200/3304 ( 66%) [12]
CEF: table : 1736/2104 ( 82%) [4]
CEF: table GSB : 468/560 ( 83%) [1]
CEF: table walks : 432/616 ( 70%) [2]
CEF: terminal fibs list : 144/328 ( 43%) [2]
CEF: test fib entry sbs : 496/680 ( 72%) [2]
CEF: up event c : 784/968 ( 80%) [2]
CEF: up event chunk : 232/416 ( 55%) [2]
CEF: v6 nd discard thrott : 4100/4192 ( 97%) [1]
CEF: v6 nd throttle chunk : 19368/19552 ( 99%) [2]
CEF: vrf : 2184/2368 ( 92%) [2]
COLL: coll rec : 368/552 ( 66%) [2]
TAL: MTRIE n08 : 1310007152/1313661944 ( 99%) [39726]
TAL: control block : 324/600 ( 54%) [3]
TAL: item list elem : 232/416 ( 55%) [2]
TAL: mtrie control block : 3560/3744 ( 95%) [2]
TAL: rtree aux : 300/576 ( 52%) [3]
TAL: rtree control block : 300/576 ( 52%) [3]
TAL: rtree nodes : 848/1032 ( 82%) [2]
TAL: tree control : 268/728 ( 36%) [5]
------------------------------------------------------------------
Memory in use/allocated Count
------------------------------------------------------------------
Totals : 2136896276/2142815648 ( 99%) [64341]
asr1002-x#sh mem sum
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 7F5AF4B0C010 7056007904 6277142588 778865316 778855216 778195964
So yes, the box clearly can digest 3M routes down to the FIB. Of course
a real world setup will have way more complex RIB side setup, with multiple
tables and stuff incoming and distilled into the actual RIB costing more
RP memory. But this was about FIB scalability, and at least to me, seeing
there clearly isn't a hard limit at 1M entries is already enough. Being
realistic, we might see IPv4+IPv6 combined break through 1M in the next
five years, but I don't expect it to break through 2M within this decade.
Anyone with more interesting show commands handy?
HTH,
Andre.
--
Cool .signatures are so 90s...
-> Andre Beck +++ ABP-RIPE +++ IBH IT-Service GmbH, Dresden <-
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list