[j-nsp] M10 FEB heap usage (RPF/route options)

[Kevin Day]

I know this is going into undocumented unsupported territory and I'm  
intentionally glossing over details that I know aren't as simple as  
I'm making them here... But, we've got an M10 that's completely run  
out of heap on the FEB after upgrading to 8.4 from 7.4.

(I dropped a few thousand v4 routes just to get things operational at  
this point, which is why it's not showing 99% here.)

FEB status:
   Temperature                        25 degrees C / 77 degrees F
   CPU utilization                     4 percent
   Interrupt utilization               0 percent
   Heap utilization                   97 percent
   Buffer utilization                 51 percent
   Total CPU DRAM                     64 MB
   Internet Processor II                 Version 1, Foundry IBM, Part  
number 9
   Start time:                           2007-12-13 13:33:51 UTC
   Uptime:                               1 day, 6 hours, 39 minutes,  
43 seconds


Realizing that's kinda high, I started poking around on the feb to see  
if I could see anything that would break down where the heap was  
allocated:


SBR(core-ams vty)# show heap
ID      Base   Total(b)    Free(b)    Used(b)   %   Name
--  --------  ---------  ---------  ---------  ---  -----------
  0    7c8ca0   50557792    1288600   49269192   97  Kernel
  1  93800000    8388608    4043972    4344636   51  Uncached

SBR(core-ams vty)# show route summary

IPv4 Route Tables:
Index         Routes     Size(b)
--------  ----------  ----------
Default       222713    15802073
1                  7         494
2                 24        1718

MPLS Route Tables:
Index         Routes     Size(b)
--------  ----------  ----------
Default            1          68

IPv6 Route Tables:
Index         Routes     Size(b)
--------  ----------  ----------
Default         1128       83108
1                 12         961
2                  4         305

According to that, my routing table shouldn't be taking much more than  
16MB + radix trees.


That didn't help much until I found "show heap 0 accounting size". It  
showed that the majority of my heap was being used by blocks sized 72,  
1612 and 5212 bytes:

SBR(core-ams vty)# show heap 0 accounting size
   Size(b)    Blocks   Total(b)
  --------  --------  ---------
        72    222670   21376320
      1612      1945    3182020
      5212      2229   11671044

That's 37 of the 50MB right there. I understand what the 72 byte  
blocks are, those should be v4 routes plus other things. I'm was a bit  
less sure about the 1612 and 5212 byte blocks are though, until I  
found this:

SBR(core-ams vty)# show piles
# of Piles  Allocated blocks  Free blocks  Free Piles  Name
----------  ----------------  -----------  ----------  ----
        189             18818           82          49  Radix tree  
attached nodes
       1945            194295          205         174  Radix tree  
unattached nodes
       2229            222736          164         133  Route options

1945 piles matches the 1945 blocks of 1612 bytes each.
2229 piles matches the 2219 blocks of 5212 bytes each.


Compare this to a nearly identical router running 7.6 and I see:

SBR(tr2 vty)# show piles
# of Piles  Allocated blocks  Free blocks  Free Piles  Name
----------  ----------------  -----------  ----------  ----
        206             20587           13           9  Radix tree  
attached nodes
       2050            204862          138         103  Radix tree  
unattached nodes
          0                 0            0           0  Route options


It made sense that whatever "route options" are was the difference. I  
found an old post here discussing SSB SDRAM usage (http://puck.nether.net/pipermail/juniper-nsp/2005-May/004275.html 
) that mentioned that RPF can require route options. I disabled rpf on  
every interface, but that did no good. I restarted the router though,  
and sure enough the allocations for route options were gone. Heap  
usage is back down to 75%.

So, my questions for you guys are:

Is this normal? I seem to remember being able to fit 500k+ routes not  
that long ago (multiple ribs). Did the size of each v4 route entry  
grow during a junos upgrade at some point?

Does this mean those of us stuck on 64MB FEB/SSB systems are going to  
have to run without RPF now or in the very near future when the size  
of a full table grows a bit more?


-- Kevin