[c-nsp] ASR 1002-X FIB scalability (was: Re: ASR-100x intro)
Pete Lumbis
alumbis at gmail.com
Tue May 28 12:06:32 EDT 2013
Since this is hardware based* you'll also need to look at how the FIB fit
down into TCAM with "show plat hard qfp act tcam resource-manager usage"
*CPP is a network processor not an ASIC like 6k, but it does rely on
similar TCAM
On Tue, May 28, 2013 at 5:45 AM, Beck, Andre <cisco-nsp at ibh.net> wrote:
> Re,
>
> On Thu, Apr 11, 2013 at 11:59:03AM +0200, Beck, Andre wrote:
> >
> > When my 1002-X box with 16GB hits the lab, before anything else, I'll set
> > it up to push BGP tables into it using bgpsimple (I also tried exabgp but
> > it gets slow as molasses as soon as you try to push some 100k prefixes)
> > and see where it goes through the roof.
>
> For starters, here is the ASR 1002-X with 16GB (atm still running 3.7.0
> consolidated as it came), flooded with a BGP table of 3.1M prefixes.
> I had to change my prefix generator to allow for prefix lengths of up
> to /27 in order to get there (trivial stateless randomized generator,
> the pigeonhole principle and all that), so it stresses the 8-8-8-8 mtrie
> even more badly. Here's some output from the ASR which took that size
> of table and FIB without a hitch:
>
> 1. The RIB side of the game
>
> asr1002-x#sh ip bgp summary
> BGP router identifier 192.168.127.12, local AS number 65533
> BGP table version is 31438353, main routing table version 31438353
> 3114278 network entries using 772340944 bytes of memory
> 3114278 path entries using 348799136 bytes of memory
> 1018812/1018812 BGP path/bestpath attribute entries using 228213888 bytes
> of memory
> 1018812 BGP AS-PATH entries using 153930440 bytes of memory
> 0 BGP route-map cache entries using 0 bytes of memory
> 0 BGP filter-list cache entries using 0 bytes of memory
> BGP using 1503284408 total bytes of memory
> BGP activity 17033337/13919059 prefixes, 17276315/14162037 paths, scan
> interval 60 secs
>
> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
> State/PfxRcd
> 192.168.111.254 4 65522 3114282 7 31438353 0 0 00:42:06
> 3114278
>
> asr1002-x#sh ip route summary
> IP routing table name is default (0x0)
> IP routing table maximum-paths is 32
> Route Source Networks Subnets Replicates Overhead Memory
> (bytes)
> connected 0 3 0 264 864
> static 0 0 0 0 0
> bgp 65533 360012 2754266 0 274056464 896912064
> External: 3114278 Internal: 0 Local: 0
> internal 97357 198402256
> Total 457369 2754269 0 274056728 1095315184
>
>
> 2. And now for the FIB stuff
>
> asr1002-x#sh ip cef summary
> IPv4 CEF is enabled for distributed and running
> VRF Default
> 3114292 prefixes (3114292/0 fwd/non-fwd)
> Table id 0x0
> Database epoch: 2 (3114292 entries at this epoch)
>
> asr1002-x#sh cef memory summary
> CEF has allocated 2136896276 bytes of memory (5919372 bytes overhead)
>
> asr1002-x#sh cef memory
> Memory in use/allocated Count
> ------------------------------------------------------------------
> ADJ: GSB : 468/560 ( 83%) [1]
> ADJ: NULL adjacency : 436/528 ( 82%) [1]
> ADJ: adj sev context : 232/416 ( 55%) [2]
> ADJ: adjacency : 3408/3776 ( 90%) [4]
> ADJ: mcprp_adj_inject_sb : 16588/16680 ( 99%) [1]
> ADJ: request resolve : 4688/4872 ( 96%) [2]
> ADJ: sevs : 432/616 ( 70%) [2]
> CEF: Brkr Updat : 9704/10440 ( 92%) [8]
> CEF: Brkr Update Rec : 312/496 ( 62%) [2]
> CEF: Brkr zone : 988/2736 ( 36%) [19]
> CEF: Broker : 5684/7432 ( 76%) [19]
> CEF: Brokers Array : 180/272 ( 66%) [1]
> CEF: EVENT msg chunks : 332/424 ( 78%) [1]
> CEF: FIB LC array : 452/544 ( 83%) [1]
> CEF: FIB LC stats array : 9860/9952 ( 99%) [1]
> CEF: FIB subtree context : 328/880 ( 37%) [6]
> CEF: FIBHWIDB : 15444/16640 ( 92%) [13]
> CEF: FIBIDB : 6292/7488 ( 84%) [13]
> CEF: IPv4 ARP throttle : 2052/2144 ( 95%) [1]
> CEF: IPv4 process : 1128/1864 ( 60%) [8]
> CEF: OCE get hash callbac : 52/144 ( 36%) [1]
> CEF: TABLE msg chunks : 804/896 ( 89%) [1]
> CEF: Table rate Monitor S : 180/456 ( 39%) [3]
> CEF: arp throttle chunk : 28168/28352 ( 99%) [2]
> CEF: cover need : 528/712 ( 74%) [2]
> CEF: cover need deagg chu : 376/560 ( 67%) [2]
> CEF: dQ elems : 360/544 ( 66%) [2]
> CEF: dQ walks : 512/696 ( 73%) [2]
> CEF: fib : 824283024/826521936 ( 99%) [24336]
> CEF: fib GSB : 23444/23904 ( 98%) [5]
> CEF: fib deps : 352/536 ( 65%) [2]
> CEF: fib loop sb : 368/552 ( 66%) [2]
> CEF: fib_fib_covered chun : 392/576 ( 68%) [2]
> CEF: fib_fib_rp_bfd_sb : 2472/2656 ( 93%) [2]
> CEF: fib_fib_sr : 4832/5936 ( 81%) [12]
> CEF: fib_fib_src_adj_sb : 368/552 ( 66%) [2]
> CEF: fib_fib_src_adj_sb_a : 320/504 ( 63%) [2]
> CEF: fib_fib_src_interfac : 344/528 ( 65%) [2]
> CEF: fib_fib_src_rr_sb : 376/560 ( 67%) [2]
> CEF: fib_fib_src_special_ : 344/528 ( 65%) [2]
> CEF: fib_head_s : 664/848 ( 78%) [2]
> CEF: fib_head_sb chunk : 376/560 ( 67%) [2]
> CEF: fib_member : 2352/2720 ( 86%) [4]
> CEF: fib_member_sb chunk : 368/552 ( 66%) [2]
> CEF: fib_rib_route_update : 5184/5368 ( 96%) [2]
> CEF: fib_table_fibswsb_de : 512/696 ( 73%) [2]
> CEF: fibhwidb table : 1048580/1048672 ( 99%) [1]
> CEF: fibidb table : 1048580/1048672 ( 99%) [1]
> CEF: fibswsb ct : 3208/3392 ( 94%) [2]
> CEF: fibswsb ctl : 1680/1864 ( 90%) [2]
> CEF: hash table : 262152/262336 ( 99%) [2]
> CEF: ipv6 feature error c : 5548/5640 ( 98%) [1]
> CEF: ipv6 feature error s : 5548/5640 ( 98%) [1]
> CEF: ipv6 not cef switche : 1228/1320 ( 93%) [1]
> CEF: ipv6 not cef switche : 1228/1320 ( 93%) [1]
> CEF: loadinf16 : 928/1112 ( 83%) [2]
> CEF: loadinfo2 : 432/616 ( 70%) [2]
> CEF: mpls long path exts : 488/672 ( 72%) [2]
> CEF: mpls path exts : 368/552 ( 66%) [2]
> CEF: nh entry context : 432/616 ( 70%) [2]
> CEF: nh entry params : 432/616 ( 70%) [2]
> CEF: non_ip entry context : 488/672 ( 72%) [2]
> CEF: pathl : 12840/13576 ( 94%) [8]
> CEF: pathl ifs : 2880/3432 ( 83%) [6]
> CEF: pathl its : 648/832 ( 77%) [2]
> CEF: pathloutputchain : 792/976 ( 81%) [2]
> CEF: paths : 13352/14088 ( 94%) [8]
> CEF: plist dq it : 488/672 ( 72%) [2]
> CEF: prefix query msg chu : 2380/2472 ( 96%) [1]
> CEF: subtree context : 2200/3304 ( 66%) [12]
> CEF: table : 1736/2104 ( 82%) [4]
> CEF: table GSB : 468/560 ( 83%) [1]
> CEF: table walks : 432/616 ( 70%) [2]
> CEF: terminal fibs list : 144/328 ( 43%) [2]
> CEF: test fib entry sbs : 496/680 ( 72%) [2]
> CEF: up event c : 784/968 ( 80%) [2]
> CEF: up event chunk : 232/416 ( 55%) [2]
> CEF: v6 nd discard thrott : 4100/4192 ( 97%) [1]
> CEF: v6 nd throttle chunk : 19368/19552 ( 99%) [2]
> CEF: vrf : 2184/2368 ( 92%) [2]
> COLL: coll rec : 368/552 ( 66%) [2]
> TAL: MTRIE n08 : 1310007152/1313661944 ( 99%) [39726]
> TAL: control block : 324/600 ( 54%) [3]
> TAL: item list elem : 232/416 ( 55%) [2]
> TAL: mtrie control block : 3560/3744 ( 95%) [2]
> TAL: rtree aux : 300/576 ( 52%) [3]
> TAL: rtree control block : 300/576 ( 52%) [3]
> TAL: rtree nodes : 848/1032 ( 82%) [2]
> TAL: tree control : 268/728 ( 36%) [5]
> ------------------------------------------------------------------
> Memory in use/allocated Count
> ------------------------------------------------------------------
>
> Totals : 2136896276/2142815648 ( 99%) [64341]
>
> asr1002-x#sh mem sum
> Head Total(b) Used(b) Free(b) Lowest(b)
> Largest(b)
> Processor 7F5AF4B0C010 7056007904 6277142588 778865316 778855216
> 778195964
>
>
> So yes, the box clearly can digest 3M routes down to the FIB. Of course
> a real world setup will have way more complex RIB side setup, with multiple
> tables and stuff incoming and distilled into the actual RIB costing more
> RP memory. But this was about FIB scalability, and at least to me, seeing
> there clearly isn't a hard limit at 1M entries is already enough. Being
> realistic, we might see IPv4+IPv6 combined break through 1M in the next
> five years, but I don't expect it to break through 2M within this decade.
>
> Anyone with more interesting show commands handy?
>
> HTH,
> Andre.
> --
> Cool .signatures are so 90s...
>
> -> Andre Beck +++ ABP-RIPE +++ IBH IT-Service GmbH, Dresden <-
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
More information about the cisco-nsp
mailing list