[c-nsp] ASR9001 BGP scaling and memory shortage

[Vladimir Troitskiy]

Hello everyone,

ASR9001 has some memory usage limits:
- 1658M for a BGP process on a RSP
- 1536M for a shared memory window on a LC
Those limits seems to be unconfigurable.

Has anybody experienced any issues with these limits on high-loaded ASR9001
boxes?
We have a surprisingly high memory usage while the typical router setup is
pretty lightweight - 4-5 full feeds (couple of upstreams and RRs). The only
probably uncommon thing is we use "Internet in a VRF" approach.

#show processes memory detail location 0/RSP0/CPU0
> Tue May 19 19:39:12.592 Ural
> JID    Text       Data       Stack      Dynamic    Dyn-Limit  Shm-Tot
>  Phy-Tot    Process
> ------ ---------- ---------- ---------- ---------- ---------- ----------
> ---------- -------
> 1054           1M         5M       516K      1485M      1658M        76M
>    1491M  bgp
>

#show memory summary location 0/0/CPU0
>
node:      node0_0_CPU0
> ------------------------------------------------------------------
> Physical Memory: 8192M total
>  Application Memory : 7988M (3811M available)
>  Image: 75M (bootram: 75M)
>  Reserved: 128M, IOMem: 0, flashfsys: 0
>  Total shared window: 1327M
>

We have already had FIB inconsistency issues due to SHMWIN exhaustion
despite the fact the total prefix amount was far from the platform limit
(4M):

> fib_mgr[184]: %OS-SHMWIN-3-ALLOC_ARENA_FAILED : SHMWIN: Failed to allocate
> new arena from the server : 'SHMWIN_SVR' detected the 'fatal' condition 'VM
> is exhausted or totally fragmented'
> fib_mgr[184]: %ROUTING-FIB-3-ASSERT_RL : FIB internal inconsistency
> detected
> fib_mgr[184]: %ROUTING-FIB-3-PD_FAIL : FIB platform error: fib_leaf_insert
> 5204 Cannot insert leaf
>

What are practical limits for BGP scaling on ASR9001 boxes? Could anyone
share a memory usage stats?
-- 
Best regards,
Vladimir Troitsky