[c-nsp] ASR 1002-X FIB scalability (was: Re: ASR-100x intro)
Beck, Andre
cisco-nsp at ibh.net
Tue May 28 05:45:11 EDT 2013
Re,
On Thu, Apr 11, 2013 at 11:59:03AM +0200, Beck, Andre wrote:
>
> When my 1002-X box with 16GB hits the lab, before anything else, I'll set
> it up to push BGP tables into it using bgpsimple (I also tried exabgp but
> it gets slow as molasses as soon as you try to push some 100k prefixes)
> and see where it goes through the roof.
For starters, here is the ASR 1002-X with 16GB (atm still running 3.7.0
consolidated as it came), flooded with a BGP table of 3.1M prefixes.
I had to change my prefix generator to allow for prefix lengths of up
to /27 in order to get there (trivial stateless randomized generator,
the pigeonhole principle and all that), so it stresses the 8-8-8-8 mtrie
even more badly. Here's some output from the ASR which took that size
of table and FIB without a hitch:
1. The RIB side of the game
asr1002-x#sh ip bgp summary
BGP router identifier 192.168.127.12, local AS number 65533
BGP table version is 31438353, main routing table version 31438353
3114278 network entries using 772340944 bytes of memory
3114278 path entries using 348799136 bytes of memory
1018812/1018812 BGP path/bestpath attribute entries using 228213888 bytes of memory
1018812 BGP AS-PATH entries using 153930440 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1503284408 total bytes of memory
BGP activity 17033337/13919059 prefixes, 17276315/14162037 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.111.254 4 65522 3114282 7 31438353 0 0 00:42:06 3114278
asr1002-x#sh ip route summary
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory (bytes)
connected 0 3 0 264 864
static 0 0 0 0 0
bgp 65533 360012 2754266 0 274056464 896912064
External: 3114278 Internal: 0 Local: 0
internal 97357 198402256
Total 457369 2754269 0 274056728 1095315184
2. And now for the FIB stuff
asr1002-x#sh ip cef summary
IPv4 CEF is enabled for distributed and running
VRF Default
3114292 prefixes (3114292/0 fwd/non-fwd)
Table id 0x0
Database epoch: 2 (3114292 entries at this epoch)
asr1002-x#sh cef memory summary
CEF has allocated 2136896276 bytes of memory (5919372 bytes overhead)
asr1002-x#sh cef memory
Memory in use/allocated Count
------------------------------------------------------------------
ADJ: GSB : 468/560 ( 83%) [1]
ADJ: NULL adjacency : 436/528 ( 82%) [1]
ADJ: adj sev context : 232/416 ( 55%) [2]
ADJ: adjacency : 3408/3776 ( 90%) [4]
ADJ: mcprp_adj_inject_sb : 16588/16680 ( 99%) [1]
ADJ: request resolve : 4688/4872 ( 96%) [2]
ADJ: sevs : 432/616 ( 70%) [2]
CEF: Brkr Updat : 9704/10440 ( 92%) [8]
CEF: Brkr Update Rec : 312/496 ( 62%) [2]
CEF: Brkr zone : 988/2736 ( 36%) [19]
CEF: Broker : 5684/7432 ( 76%) [19]
CEF: Brokers Array : 180/272 ( 66%) [1]
CEF: EVENT msg chunks : 332/424 ( 78%) [1]
CEF: FIB LC array : 452/544 ( 83%) [1]
CEF: FIB LC stats array : 9860/9952 ( 99%) [1]
CEF: FIB subtree context : 328/880 ( 37%) [6]
CEF: FIBHWIDB : 15444/16640 ( 92%) [13]
CEF: FIBIDB : 6292/7488 ( 84%) [13]
CEF: IPv4 ARP throttle : 2052/2144 ( 95%) [1]
CEF: IPv4 process : 1128/1864 ( 60%) [8]
CEF: OCE get hash callbac : 52/144 ( 36%) [1]
CEF: TABLE msg chunks : 804/896 ( 89%) [1]
CEF: Table rate Monitor S : 180/456 ( 39%) [3]
CEF: arp throttle chunk : 28168/28352 ( 99%) [2]
CEF: cover need : 528/712 ( 74%) [2]
CEF: cover need deagg chu : 376/560 ( 67%) [2]
CEF: dQ elems : 360/544 ( 66%) [2]
CEF: dQ walks : 512/696 ( 73%) [2]
CEF: fib : 824283024/826521936 ( 99%) [24336]
CEF: fib GSB : 23444/23904 ( 98%) [5]
CEF: fib deps : 352/536 ( 65%) [2]
CEF: fib loop sb : 368/552 ( 66%) [2]
CEF: fib_fib_covered chun : 392/576 ( 68%) [2]
CEF: fib_fib_rp_bfd_sb : 2472/2656 ( 93%) [2]
CEF: fib_fib_sr : 4832/5936 ( 81%) [12]
CEF: fib_fib_src_adj_sb : 368/552 ( 66%) [2]
CEF: fib_fib_src_adj_sb_a : 320/504 ( 63%) [2]
CEF: fib_fib_src_interfac : 344/528 ( 65%) [2]
CEF: fib_fib_src_rr_sb : 376/560 ( 67%) [2]
CEF: fib_fib_src_special_ : 344/528 ( 65%) [2]
CEF: fib_head_s : 664/848 ( 78%) [2]
CEF: fib_head_sb chunk : 376/560 ( 67%) [2]
CEF: fib_member : 2352/2720 ( 86%) [4]
CEF: fib_member_sb chunk : 368/552 ( 66%) [2]
CEF: fib_rib_route_update : 5184/5368 ( 96%) [2]
CEF: fib_table_fibswsb_de : 512/696 ( 73%) [2]
CEF: fibhwidb table : 1048580/1048672 ( 99%) [1]
CEF: fibidb table : 1048580/1048672 ( 99%) [1]
CEF: fibswsb ct : 3208/3392 ( 94%) [2]
CEF: fibswsb ctl : 1680/1864 ( 90%) [2]
CEF: hash table : 262152/262336 ( 99%) [2]
CEF: ipv6 feature error c : 5548/5640 ( 98%) [1]
CEF: ipv6 feature error s : 5548/5640 ( 98%) [1]
CEF: ipv6 not cef switche : 1228/1320 ( 93%) [1]
CEF: ipv6 not cef switche : 1228/1320 ( 93%) [1]
CEF: loadinf16 : 928/1112 ( 83%) [2]
CEF: loadinfo2 : 432/616 ( 70%) [2]
CEF: mpls long path exts : 488/672 ( 72%) [2]
CEF: mpls path exts : 368/552 ( 66%) [2]
CEF: nh entry context : 432/616 ( 70%) [2]
CEF: nh entry params : 432/616 ( 70%) [2]
CEF: non_ip entry context : 488/672 ( 72%) [2]
CEF: pathl : 12840/13576 ( 94%) [8]
CEF: pathl ifs : 2880/3432 ( 83%) [6]
CEF: pathl its : 648/832 ( 77%) [2]
CEF: pathloutputchain : 792/976 ( 81%) [2]
CEF: paths : 13352/14088 ( 94%) [8]
CEF: plist dq it : 488/672 ( 72%) [2]
CEF: prefix query msg chu : 2380/2472 ( 96%) [1]
CEF: subtree context : 2200/3304 ( 66%) [12]
CEF: table : 1736/2104 ( 82%) [4]
CEF: table GSB : 468/560 ( 83%) [1]
CEF: table walks : 432/616 ( 70%) [2]
CEF: terminal fibs list : 144/328 ( 43%) [2]
CEF: test fib entry sbs : 496/680 ( 72%) [2]
CEF: up event c : 784/968 ( 80%) [2]
CEF: up event chunk : 232/416 ( 55%) [2]
CEF: v6 nd discard thrott : 4100/4192 ( 97%) [1]
CEF: v6 nd throttle chunk : 19368/19552 ( 99%) [2]
CEF: vrf : 2184/2368 ( 92%) [2]
COLL: coll rec : 368/552 ( 66%) [2]
TAL: MTRIE n08 : 1310007152/1313661944 ( 99%) [39726]
TAL: control block : 324/600 ( 54%) [3]
TAL: item list elem : 232/416 ( 55%) [2]
TAL: mtrie control block : 3560/3744 ( 95%) [2]
TAL: rtree aux : 300/576 ( 52%) [3]
TAL: rtree control block : 300/576 ( 52%) [3]
TAL: rtree nodes : 848/1032 ( 82%) [2]
TAL: tree control : 268/728 ( 36%) [5]
------------------------------------------------------------------
Memory in use/allocated Count
------------------------------------------------------------------
Totals : 2136896276/2142815648 ( 99%) [64341]
asr1002-x#sh mem sum
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 7F5AF4B0C010 7056007904 6277142588 778865316 778855216 778195964
So yes, the box clearly can digest 3M routes down to the FIB. Of course
a real world setup will have way more complex RIB side setup, with multiple
tables and stuff incoming and distilled into the actual RIB costing more
RP memory. But this was about FIB scalability, and at least to me, seeing
there clearly isn't a hard limit at 1M entries is already enough. Being
realistic, we might see IPv4+IPv6 combined break through 1M in the next
five years, but I don't expect it to break through 2M within this decade.
Anyone with more interesting show commands handy?
HTH,
Andre.
--
Cool .signatures are so 90s...
-> Andre Beck +++ ABP-RIPE +++ IBH IT-Service GmbH, Dresden <-
More information about the cisco-nsp
mailing list