[j-nsp] Apparent m40 RPD memory leak

[Jared Gillis]

Hi all,

I've been scratching my head over a problem with steadily increasing memory usage by rpd on our m40 routers.
Two of our core routers have been exhibiting this problem for quite some time. Based on our graphing of memory utilization, we see the usage climb up steadily until it maxes out somewhere between 90 and 100%. Upon investigation on the router, we find that rpd is using all of this additional memory. Requesting a routing restart flushes the memory, but it starts climbing back up just like before.

These two routers are deployed side by side and have been running problem-free for almost 3 years. They were running 6.2R1.5 and 6.3R1.3 respectively.
After spending many months trying everything we could think of to resolve the problem, we decided to try upgrading to the next major rev. The router running 6.2R1.5 was upgraded to 7.5R4.4 about a month and a half ago, but we are *still* seeing the apparent memory leak.
Here are our graphs for the last year:

http://www.sonic.net/jared/gw3-mem.png
This is the router that was upgraded to 7.5. The drops in memory utilization before that were routing restarts, and you can see how extremely rapidly the memory utilization climbed back up.

http://www.sonic.net/jared/gw4-mem.png
This is the other router, still running the same 6.3 code. It's memory utilization climbs much less quickly, but it is still a problem.

Both routers are transit/edge routers, and thus handle a full table, and both are running quite a few ibgp and ebgp processes (about 20 of each). The only other protocol they're running is OSPF. They're not running anything that I would consider to be unusual, filters and routing policies for TE/BCP, etc.

We have an m10i that was running 6.x code, and was upgraded to 7.5 at approximately the same time as gw3, but never exhibited this problem on either code release.

Does anyone have any ideas on what on earth could be causing this?

-- 
Jared Gillis - jared at corp.sonic.net       Sonic.net, Inc.
Network Operations                        2260 Apollo Way
707.522.1000 (Voice)                      Santa Rosa, CA 95407
707.547.3400 (Support)                    http://www.sonic.net/