[j-nsp] FE Memory Upgrade

[Eric Van Tol]

Actually, this is how JTAC explained it, in the context of DRAM:

Total DRAM
-----------
ukernel heap (55MB) + uncached heap (8MB) + PFE code/data seg(1MB) =
64MB

We had 99% utilization of ukernel heap.  Here's the breakdown:

uKernel Heap
------------
Route   : 42   MB
NH      :  0.5 MB
Firewall:  1.5 MB
Shadow  :  8   MB
Miscell.:  3   MB (for all other stuff)
Total   : ~55 MB 

SSB1(xxxxxxx vty)# show route summary

IPv4 Route Tables:
Index         Routes     Size(b)
--------  ----------  ----------
Default       160307    11373687 <------- main
1                  4         281
2             159909    11345021 <------- logical
3                 18        1289
4                 18        1289

MPLS Route Tables:
Index         Routes     Size(b)
--------  ----------  ----------
Default            4         281
2                  1          68

IPV6 Route Tables:
Index         Routes     Size(b)
--------  ----------  ----------
Default            3         221
1                  5         389
2                  3         221
3                  3         221
4                  3         221

  Thus using significant route counts:
   RT_MEM_1 = (160307 + 159909) * (72 + 24)
            = ~30MB

We are using RPF checking. This leads to more overhead in the form of
route options (size 40 bytes)

SSB1(xxxxxxxxx vty)# show piles 
# of Piles  Allocated blocks  Free blocks  Name
----------  ----------------  -----------  ----
        318             28082         3718  Radix tree attached nodes
       2824            277267         5133  Radix tree unattached nodes
       1653            160311         4989  Route options

    RT_MEM_2 = (160311 + 4989) * (40)
             = ~6.3 MB

The Radix tree, maintianed in the PFE, also consumes memory.

SSB1(xxxxxx vty)# show radix statistics
Type    # of blocks    total size
----    -----------    ----------
root             19           684
attach        28086        561720
noattach     277275       4436400
totals       305380       4998804
---------------------------------
Error type                  count
----------                  -----
Illegal Init                    0
Internal Error                  0
Allocation Failed               0

   RT_MEM_3 = 4998804
            = ~5MB

   Total route mem = RT_MEM_1 + RT_MEM_2 + RT_MEM_3
                   =   30     +    6.3   +   5
                   =  ~42MB

I wish I could take credit for gathering all this information, but it
was JTAC who provided it to me.  While I'm sure that the SRAM was also
severely affected, the exact problem was definitely with the amount of
DRAM in the SSB.  Killing the logical-router in this M20 (which held a
duplicate copy of a full table) had brought the DRAM usage down to 70%,
which does seem to fit somewhat with your clarification, as that table
was also a "FIB" with active next-hop information.

-eric

-----Original Message-----
From: Richard A Steenbergen [mailto:ras at e-gerbil.net] 
Sent: Tuesday, May 10, 2005 3:15 PM
To: Eric Van Tol
Cc: Sabri Berisha; info at beprojects.com; juniper-nsp at puck.nether.net
Subject: Re: [j-nsp] FE Memory Upgrade

On Tue, May 10, 2005 at 12:04:07PM -0400, Eric Van Tol wrote:
> I have no M7i experience, but I can tell you that an M20 with an SSB-E
> and 64MB DRAM/8MB SRAM could only handle about 500K routes in the Rib
> before it started to issue allocation errors.  768MB on the RE and
16MB
> SRAM on the CFEB will do you fine, as long as you're not going to use
it
> to create 15 logical-routers, all with a copy of a full BGP table ;-)
> 
> I received a good explanation from Juniper about how the memory
> allocation in the SSB (or CFEB in your case) is handled (paraphrased
> from JTAC):

That would be your SRAM that ran out, not the SSB DRAM. The DRAM is used

to hold the next-hop table, which can run out if you have a large amount

of arp entries on an ethernet interface for example.

As it was described to me, the nhdb L2 descriptor size is hard coded as
a 
percentage of your DRAM. On a 64MB DRAM model, this comes out to just
over 
50k:


SFM2(xxx.xxx vty)# show nhdb zones
Chip  Start   Size   Rsvd   Used/Hi Water/Total  Size  Name
----  -----  -----  -----  --------------------  ----  ----
   0  20000  01000  00000               0/0/512     8  Multicast RTP
   0  21000  02400  00000              0/0/9216     1  Multicast Lists
   0  23400  04400  00006         176/180/17408     1  Next-Hop Entries
   0  27800  18800  0004b         856/885/50176     2  L2 Descriptors
                                  ^^^^^^^^^^^^^

If you have 50k arp entries on your system, it can exhaust the max L2 
descriptors. The Juniper answer is to buy a new switch board which has 
128MB of DRAM on it. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>
http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1
2CBC)