[c-nsp] ASR 1002-X FIB scalability (was: Re: ASR-100x intro)

Pete Lumbis alumbis at gmail.com
Tue May 28 12:06:32 EDT 2013


Since this is hardware based* you'll also need to look at how the FIB fit
down into TCAM with "show plat hard qfp act tcam resource-manager usage"

*CPP is a network processor not an ASIC like 6k, but it does rely on
similar TCAM


On Tue, May 28, 2013 at 5:45 AM, Beck, Andre <cisco-nsp at ibh.net> wrote:

> Re,
>
> On Thu, Apr 11, 2013 at 11:59:03AM +0200, Beck, Andre wrote:
> >
> > When my 1002-X box with 16GB hits the lab, before anything else, I'll set
> > it up to push BGP tables into it using bgpsimple (I also tried exabgp but
> > it gets slow as molasses as soon as you try to push some 100k prefixes)
> > and see where it goes through the roof.
>
> For starters, here is the ASR 1002-X with 16GB (atm still running 3.7.0
> consolidated as it came), flooded with a BGP table of 3.1M prefixes.
> I had to change my prefix generator to allow for prefix lengths of up
> to /27 in order to get there (trivial stateless randomized generator,
> the pigeonhole principle and all that), so it stresses the 8-8-8-8 mtrie
> even more badly. Here's some output from the ASR which took that size
> of table and FIB without a hitch:
>
> 1. The RIB side of the game
>
> asr1002-x#sh ip bgp summary
> BGP router identifier 192.168.127.12, local AS number 65533
> BGP table version is 31438353, main routing table version 31438353
> 3114278 network entries using 772340944 bytes of memory
> 3114278 path entries using 348799136 bytes of memory
> 1018812/1018812 BGP path/bestpath attribute entries using 228213888 bytes
> of memory
> 1018812 BGP AS-PATH entries using 153930440 bytes of memory
> 0 BGP route-map cache entries using 0 bytes of memory
> 0 BGP filter-list cache entries using 0 bytes of memory
> BGP using 1503284408 total bytes of memory
> BGP activity 17033337/13919059 prefixes, 17276315/14162037 paths, scan
> interval 60 secs
>
> Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down
>  State/PfxRcd
> 192.168.111.254 4        65522 3114282       7 31438353    0    0 00:42:06
>  3114278
>
> asr1002-x#sh ip route summary
> IP routing table name is default (0x0)
> IP routing table maximum-paths is 32
> Route Source    Networks    Subnets     Replicates  Overhead    Memory
> (bytes)
> connected       0           3           0           264         864
> static          0           0           0           0           0
> bgp 65533       360012      2754266     0           274056464   896912064
>   External: 3114278 Internal: 0 Local: 0
> internal        97357                                           198402256
> Total           457369      2754269     0           274056728   1095315184
>
>
> 2. And now for the FIB stuff
>
> asr1002-x#sh ip cef summary
> IPv4 CEF is enabled for distributed and running
> VRF Default
>  3114292 prefixes (3114292/0 fwd/non-fwd)
>  Table id 0x0
>  Database epoch:        2 (3114292 entries at this epoch)
>
> asr1002-x#sh cef memory summary
> CEF has allocated 2136896276 bytes of memory (5919372 bytes overhead)
>
> asr1002-x#sh cef memory
>   Memory                          in use/allocated         Count
>   ------------------------------------------------------------------
>   ADJ: GSB                  :        468/560        ( 83%) [1]
>   ADJ: NULL adjacency       :        436/528        ( 82%) [1]
>   ADJ: adj sev context      :        232/416        ( 55%) [2]
>   ADJ: adjacency            :       3408/3776       ( 90%) [4]
>   ADJ: mcprp_adj_inject_sb  :      16588/16680      ( 99%) [1]
>   ADJ: request resolve      :       4688/4872       ( 96%) [2]
>   ADJ: sevs                 :        432/616        ( 70%) [2]
>   CEF: Brkr Updat           :       9704/10440      ( 92%) [8]
>   CEF: Brkr Update Rec      :        312/496        ( 62%) [2]
>   CEF: Brkr zone            :        988/2736       ( 36%) [19]
>   CEF: Broker               :       5684/7432       ( 76%) [19]
>   CEF: Brokers Array        :        180/272        ( 66%) [1]
>   CEF: EVENT msg chunks     :        332/424        ( 78%) [1]
>   CEF: FIB LC array         :        452/544        ( 83%) [1]
>   CEF: FIB LC stats array   :       9860/9952       ( 99%) [1]
>   CEF: FIB subtree context  :        328/880        ( 37%) [6]
>   CEF: FIBHWIDB             :      15444/16640      ( 92%) [13]
>   CEF: FIBIDB               :       6292/7488       ( 84%) [13]
>   CEF: IPv4 ARP throttle    :       2052/2144       ( 95%) [1]
>   CEF: IPv4 process         :       1128/1864       ( 60%) [8]
>   CEF: OCE get hash callbac :         52/144        ( 36%) [1]
>   CEF: TABLE msg chunks     :        804/896        ( 89%) [1]
>   CEF: Table rate Monitor S :        180/456        ( 39%) [3]
>   CEF: arp throttle chunk   :      28168/28352      ( 99%) [2]
>   CEF: cover need           :        528/712        ( 74%) [2]
>   CEF: cover need deagg chu :        376/560        ( 67%) [2]
>   CEF: dQ elems             :        360/544        ( 66%) [2]
>   CEF: dQ walks             :        512/696        ( 73%) [2]
>   CEF: fib                  :  824283024/826521936  ( 99%) [24336]
>   CEF: fib GSB              :      23444/23904      ( 98%) [5]
>   CEF: fib deps             :        352/536        ( 65%) [2]
>   CEF: fib loop sb          :        368/552        ( 66%) [2]
>   CEF: fib_fib_covered chun :        392/576        ( 68%) [2]
>   CEF: fib_fib_rp_bfd_sb    :       2472/2656       ( 93%) [2]
>   CEF: fib_fib_sr           :       4832/5936       ( 81%) [12]
>   CEF: fib_fib_src_adj_sb   :        368/552        ( 66%) [2]
>   CEF: fib_fib_src_adj_sb_a :        320/504        ( 63%) [2]
>   CEF: fib_fib_src_interfac :        344/528        ( 65%) [2]
>   CEF: fib_fib_src_rr_sb    :        376/560        ( 67%) [2]
>   CEF: fib_fib_src_special_ :        344/528        ( 65%) [2]
>   CEF: fib_head_s           :        664/848        ( 78%) [2]
>   CEF: fib_head_sb chunk    :        376/560        ( 67%) [2]
>   CEF: fib_member           :       2352/2720       ( 86%) [4]
>   CEF: fib_member_sb chunk  :        368/552        ( 66%) [2]
>   CEF: fib_rib_route_update :       5184/5368       ( 96%) [2]
>   CEF: fib_table_fibswsb_de :        512/696        ( 73%) [2]
>   CEF: fibhwidb table       :    1048580/1048672    ( 99%) [1]
>   CEF: fibidb table         :    1048580/1048672    ( 99%) [1]
>   CEF: fibswsb ct           :       3208/3392       ( 94%) [2]
>   CEF: fibswsb ctl          :       1680/1864       ( 90%) [2]
>   CEF: hash table           :     262152/262336     ( 99%) [2]
>   CEF: ipv6 feature error c :       5548/5640       ( 98%) [1]
>   CEF: ipv6 feature error s :       5548/5640       ( 98%) [1]
>   CEF: ipv6 not cef switche :       1228/1320       ( 93%) [1]
>   CEF: ipv6 not cef switche :       1228/1320       ( 93%) [1]
>   CEF: loadinf16            :        928/1112       ( 83%) [2]
>   CEF: loadinfo2            :        432/616        ( 70%) [2]
>   CEF: mpls long path exts  :        488/672        ( 72%) [2]
>   CEF: mpls path exts       :        368/552        ( 66%) [2]
>   CEF: nh entry context     :        432/616        ( 70%) [2]
>   CEF: nh entry params      :        432/616        ( 70%) [2]
>   CEF: non_ip entry context :        488/672        ( 72%) [2]
>   CEF: pathl                :      12840/13576      ( 94%) [8]
>   CEF: pathl ifs            :       2880/3432       ( 83%) [6]
>   CEF: pathl its            :        648/832        ( 77%) [2]
>   CEF: pathloutputchain     :        792/976        ( 81%) [2]
>   CEF: paths                :      13352/14088      ( 94%) [8]
>   CEF: plist dq it          :        488/672        ( 72%) [2]
>   CEF: prefix query msg chu :       2380/2472       ( 96%) [1]
>   CEF: subtree context      :       2200/3304       ( 66%) [12]
>   CEF: table                :       1736/2104       ( 82%) [4]
>   CEF: table GSB            :        468/560        ( 83%) [1]
>   CEF: table walks          :        432/616        ( 70%) [2]
>   CEF: terminal fibs list   :        144/328        ( 43%) [2]
>   CEF: test fib entry sbs   :        496/680        ( 72%) [2]
>   CEF: up event c           :        784/968        ( 80%) [2]
>   CEF: up event chunk       :        232/416        ( 55%) [2]
>   CEF: v6 nd discard thrott :       4100/4192       ( 97%) [1]
>   CEF: v6 nd throttle chunk :      19368/19552      ( 99%) [2]
>   CEF: vrf                  :       2184/2368       ( 92%) [2]
>   COLL: coll rec            :        368/552        ( 66%) [2]
>   TAL: MTRIE n08            : 1310007152/1313661944 ( 99%) [39726]
>   TAL: control block        :        324/600        ( 54%) [3]
>   TAL: item list elem       :        232/416        ( 55%) [2]
>   TAL: mtrie control block  :       3560/3744       ( 95%) [2]
>   TAL: rtree aux            :        300/576        ( 52%) [3]
>   TAL: rtree control block  :        300/576        ( 52%) [3]
>   TAL: rtree nodes          :        848/1032       ( 82%) [2]
>   TAL: tree control         :        268/728        ( 36%) [5]
>   ------------------------------------------------------------------
>   Memory                          in use/allocated         Count
>   ------------------------------------------------------------------
>
>   Totals                    : 2136896276/2142815648 ( 99%) [64341]
>
> asr1002-x#sh mem sum
>                 Head    Total(b)     Used(b)     Free(b)   Lowest(b)
>  Largest(b)
> Processor  7F5AF4B0C010   7056007904   6277142588   778865316   778855216
>   778195964
>
>
> So yes, the box clearly can digest 3M routes down to the FIB. Of course
> a real world setup will have way more complex RIB side setup, with multiple
> tables and stuff incoming and distilled into the actual RIB costing more
> RP memory. But this was about FIB scalability, and at least to me, seeing
> there clearly isn't a hard limit at 1M entries is already enough. Being
> realistic, we might see IPv4+IPv6 combined break through 1M in the next
> five years, but I don't expect it to break through 2M within this decade.
>
> Anyone with more interesting show commands handy?
>
> HTH,
> Andre.
> --
>                     Cool .signatures are so 90s...
>
> -> Andre Beck    +++ ABP-RIPE +++      IBH IT-Service GmbH, Dresden <-
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>


More information about the cisco-nsp mailing list