[j-nsp] Juniper M20 memory problem
Josef Buchsteiner
josefb at juniper.net
Mon Jun 11 03:10:56 EDT 2007
Nicolaj,
I was more trying to understand why the kernel is complaining
of
"Nexthop index allocation failed: regular index space
exhausted"
In this version the regular index space from kernels
perspective is 256k. Before you reach such limit you are
running out of resources either on the FPC due to Descriptor
space shortage or due to DRAM Memory shortage on the S-Board
so this should never we the bottleneck in the version you use.
Anyway, I did not got to an answer here and maybe I would need
to complete message log and or remote login. The number of
rtnexthops is 17411 entries as you can see from the virtual
memory point of view.
From the data below we can see that the mas next-hop entries
was 8106 unicast next-hops on both FPCs. You had double of
those entries since you had an aggregate with 2 member links
and you can see already the number of next-hops and resources
depends on the number of memberlinks of an aggregate. Also
the L2 Descriptor space was with 32532 entries below the
limit. Any increase you would need to have an Enhanced FPC.
SSB1(re0 vty)# show nhdb zones
Chip Start Size Rsvd Used/Hi Water/Total Size Name
---- ----- ----- ----- -------------------- ---- ----
0 20000 01000 00000 0/0/512 8 Multicast RTP
0 21000 02400 00000 0/0/9216 1 Multicast Lists
0 23400 04400 00006 7202/8106/17408 1 Next-Hop Entries
0 27800 18800 0004b 28928/32532/50176 2 L2 Descriptors
0 40400 001ff 00000 3/3/63 8 L2 Programs
1 20000 01000 00000 0/0/512 8 Multicast RTP
1 21000 02400 00000 0/0/9216 1 Multicast Lists
1 23400 04400 00006 0/8106/17408 1 Next-Hop Entries
1 27800 18800 0004b 120/32532/50176 2 L2 Descriptors
1 40400 001ff 00000 2/3/63 8 L2 Programs
If you have arp entries only with vlans and without vlans here
are quick summary on how far you can go depending on the FPC
Type. Those numbers are per FPC and not per system and the DRAM
memory of the S-Board is the other part which could limit the
numbers.
arp entries | ether | with vlan
------------+--------+------------
FPC | 16750 | 13060
| |
E-FPC | 54272 | 40722
| |
E2-FPC | 61183 | 61183
M10i/M7i | |
Since you were not running into those merits you indicated that
you were running out of memory resources on the S-Board and
your utilization is already about 76%. As a guideline usually
you should look at the details between 80-90% where the
resources are going to better manage this for future and
plan the upgrade or changes. Any transient change might need
some high peeks of resources and you should have some room left
to make such kind of operations. Once you are at 90% then I
usually suggest for an immediate actions in short time frames.
In order to illustrate how an aggregate is build and how much
memory we need here an example
Route -> Aggregate next-hop ---> unicast next-hop
|
+-----------> unicast next-hop
route entry:
============
ipv4 route is 68byte + 4byte(ipv4) = 72byte
(If you have uRPF enabled you can add 52byte to each route)
unicast nexthop:
================
each of them is 348 byte which includes the max possible
L2 header
Aggregate next-hop:
===================
92 + (x * 40) where x is the number of links in an aggregate
which is in your case 172byte.
So in your case with 8k of arp entries and you create an
aggregate of 2 member links you need to following more resources
on memory space. You double the unicast entries and you add
an additional aggregate next-hops for each route.
172byte * 8106 = 1,39MB
384byte * 8106 = 3.11MB
So in total about 4,5MB more.
If you have the enhanced SSB-E with 256M then this will never be
the restriction and you would need to upgrade the FPCs to E-FPC
and then you can have 54k entries on each FPCs and getting
really high numbers of arp entries. Of course of you have then
100k or 150k on arp entries you would need to think on how to
deal with arp-refresh and so on and the regular resources you
have on the RE. e.g. RE3.0 is recommended.
hope this helps a bit.
Josef
Sunday, June 10, 2007, 3:24:44 PM, you wrote:
NK>
NK>
NK> Josef Buchsteiner wrote:
NK>
NK> Hi Josef,
NK>
>> show version
>> show system virtual-memory
NK>
NK> the output is below. But meanwhile of course I had to fix the problem
NK> and therefore moved some Gateway IPs to a different router to decrease
NK> the arp count from 7200 to 5611 currently. I had to reboot the SSB-E to
NK> free the memory but at least it is working now.
NK>
NK> nico at re0> show version
NK> Hostname: re0
NK> Model: m20
NK> JUNOS Base OS boot [7.6R1.10]
NK> JUNOS Base OS Software Suite [7.6R1.10]
NK> JUNOS Kernel Software Suite [7.6R1.10]
NK> JUNOS Packet Forwarding Engine Support (M20/M40) [7.6R1.10]
NK> JUNOS Routing Software Suite [7.6R1.10]
NK> JUNOS Online Documentation [7.6R1.10]
NK> JUNOS Crypto Software Suite [7.6R1.10]
NK>
NK>
NK>
NK> nico at re0> show system virtual-memory
NK> Memory statistics by bucket size
NK> Size In Use Free Requests HighWater Couldfree
NK> 16 196393 183767 2355256898 1280 1140495
NK> 32 30503 249817 1418360018 640 298805
NK> 64 460785 515407 1405666746 320 1857754
NK> 128 20788 244748 686148930 160 944141
NK> 256 21129 11255 992196792 80 1145025
NK> 512 113 15 340207290 40 0
NK> 1K 81 7 105736 20 52
NK> 2K 195 87 17036 10 3309
NK> 4K 17 10 55228 5 1744
NK> 8K 73 5 87868 5 669
NK> 16K 8 0 15 5 0
NK> 32K 12 0 2281 5 0
NK> 64K 18 0 18 5 0
NK> 128K 5 0 5 5 0
NK> 256K 2 0 2 5 0
NK> 512K 1 0 1 5 0
NK>
NK> Memory usage type by bucket size
NK> Size Type(s)
NK> 16 uc_devlist, nexusdev, iftable, temp, devbuf, atexit, COS, BPF,
NK> DEVFS mount, DEVFS node, vnodes, mount, pcb, soname,
NK> proc-args, kld,
NK> MD disk, rman, ATA generic, bus, sysctl, pfestat, ifstate,
NK> pfe_ipc,
NK> mkey, rtable, ifmaddr, ipfw, rnode
NK> 32 atkbddev, dirrem, mkdir, diradd, freefile, freefrag, indirdep,
NK> bmsafemap, newblk, temp, devbuf, tseg_qent, COS, vnodes,
NK> cluster_save buffer, pcb, soname, proc-args, sigio, kld, Gzip
NK> trees,
NK> taskqueue, SWAP, eventhandler, bus, sysctl, uidinfo, subproc,
NK> pgrp,
NK> ippool, pfestat, itable32, ifstate, pfe_ipc, mkey, rtable,
NK> ifmaddr,
NK> ipfw, ifdevice, iflist, rnode, rtnexthop
NK> 64 isadev, ip6ndp, MFS node, allocindir, allocdirect, pagedep, temp,
NK> devbuf, lockf, COS, NULLFS hash, DEVFS name, vnodes,
NK> cluster_save buffer, vfscache, pcb, soname, proc-args, file,
NK> AR driver, AD driver, Gzip trees, rman, eventhandler, bus,
NK> sysctl,
NK> subproc, pfestat, pic, ifstate, pfe_ipc, mkey, ifaddr,
NK> rtable, ipfw,
NK> rnode
NK> 128 ZONE, freeblks, inodedep, temp, devbuf, zombie, COS, BPF, DEVFS node,
NK> vnodes, mount, cluster_save buffer, pcb, soname, proc-args, ttys,
NK> dev_t, timecounter, kld, Gzip trees, ISOFS node, bus,
NK> uidinfo, cred,
NK> session, pfestat, pic, itable16, ifstate, pfe_ipc, mkey, rtable,
NK> ipfw, ifstat, metrics, rtnexthop, iffamily
NK> 256 iflogical, iftable, MFS node, FFS node, newblk, temp, devbuf,
NK> NFS daemon, vnodes, proc-args, kqueue, file desc, Gzip trees,
NK> bus,
NK> subproc, itable16, ifstate, pfe_ipc, sysctl, rtable, ipfw, rnode,
NK> rtnexthop, iffamily
NK> 512 UFS mount, temp, devbuf, mount, BIO buffer, ptys, ttys, file desc,
NK> AR driver, Gzip trees, ISOFS mount, msg, ioctlops, ATA
NK> generic, bus,
NK> proc, pfestat, lr, ifstate, pfe_ipc, rtable, ipfw, ifstat,
NK> rtnexthop
NK> 1K iftable, temp, devbuf, NQNFS Lease, COS, BIO buffer, kqueue,
NK> file desc, kld, AD driver, Gzip trees, sem, MD disk, bus,
NK> ifstate,
NK> pfe_ipc, mkey, ipfw
NK> 2K uc_devlist, iflogical, UFS mount, temp, devbuf, BIO buffer, pcb,
NK> AR driver, Gzip trees, ioctlops, bus, pfe_ipc, ipfw, ifstat,
NK> rcache
NK> 4K memdesc, iftable, UFS mount, temp, devbuf, kld, Gzip trees, sem, msg,
NK> bus, iffclassstat
NK> 8K iflogical, indirdep, temp, devbuf, syncache, Gzip trees
NK> 16K UFS mount, temp, devbuf, shm, msg
NK> 32K pagedep, devbuf, BPF, kld, Gzip trees
NK> 64K VM pgdata, devbuf, MSDOSFS mount
NK> 128K UFS ihash, inodedep, NFS hash, kld, ISOFS mount
NK> 256K mbuf, vfscache
NK> 512K SWAP
NK>
NK> Memory statistics by type Type Kern
NK> Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
NK> isadev 13 1K 1K127696K 13 0 0 64
NK> atkbddev 2 1K 1K127696K 2 0 0 32
NK> uc_devlist 24 3K 3K127696K 24 0 0 16,2K
NK> nexusdev 3 1K 1K127696K 3 0 0 16
NK> memdesc 1 4K 4K127696K 1 0 0 4K
NK> mbuf 1 152K 152K127696K 1 0 0 256K
NK> iflogical 159 558K 576K127696K 543 0 0 256,2K,8K
NK> iftable 16 9K 9K127696K 28 0 0 16,256,1K,4K
NK> ip6ndp 0 0K 1K127696K 2 0 0 64
NK> ZONE 15 2K 2K127696K 15 0 0 128
NK> VM pgdata 1 64K 64K127696K 1 0 0 64K
NK> UFS mount 15 41K 41K127696K 15 0 0
NK> 512,2K,4K,16K
NK> UFS ihash 1 128K 128K127696K 1 0 0 128K
NK> MFS node 261 65K 72K127696K 11312 0 0 64,256
NK> FFS node 1212 303K 304K127696K 10321 0 0 256
NK> dirrem 0 0K 4K127696K 41400 0 0 32
NK> mkdir 0 0K 1K127696K 8 0 0 32
NK> diradd 0 0K 2K127696K 41433 0 0 32
NK> freefile 0 0K 1K127696K 7302 0 0 32
NK> freeblks 0 0K 1K127696K 8682 0 0 128
NK> freefrag 0 0K 3K127696K 135820 0 0 32
NK> allocindir 0 0K 248K127696K 406886 0 0 64
NK> indirdep 0 0K 33K127696K 166778 0 0 32,8K
NK> allocdirect 0 0K 4K127696K 232268 0 0 64
NK> bmsafemap 0 0K 1K127696K 146306 0 0 32
NK> newblk 1 1K 1K127696K 639155 0 0 32,256
NK> inodedep 1 128K 137K127696K 53025 0 0 128,128K
NK> pagedep 1 32K 33K127696K 5813 0 0 64,32K
NK> temp 1337 94K 99K127696K 337591 0 0
NK> 16,32,64,128,256,512,1K,2K,4K,8K,16K
NK> devbuf 16825 2350K 2671K127696K 1042931 0 0
NK> 16,32,64,128,256,512,1K,2K,4K,8K,16K,32K,64K
NK> lockf 42 3K 3K127696K 6112563 0 0 64
NK> atexit 1 1K 1K127696K 1 0 0 16
NK> zombie 0 0K 1K127696K 143910 0 0 128
NK> NFS hash 1 128K 128K127696K 1 0 0 128K
NK> NQNFS Lease 1 1K 1K127696K 1 0 0 1K
NK> NFS daemon 1 1K 1K127696K 1 0 0 256
NK> syncache 1 8K 8K127696K 1 0 0 8K
NK> tseg_qent 0 0K 1K127696K 1452 0 0 32
NK> COS 87 63K 65K127696K 349 0 0
NK> 16,32,64,128,1K
NK> BPF 403 7K 69K127696K 524 0 0 16,128,32K
NK> MSDOSFS mount 1 64K 64K127696K 1 0 0 64K
NK> NULLFS hash 1 1K 1K127696K 1 0 0 64
NK> DEVFS mount 2 1K 1K127696K 2 0 0 16
NK> DEVFS name 493 31K 31K127696K 493 0 0 64
NK> DEVFS node 479 59K 59K127696K 517 0 0 16,128
NK> vnodes 28 7K 7K127696K 429 0 0
NK> 16,32,64,128,256
NK> mount 15 8K 8K127696K 17 0 0 16,128,512
NK> cluster_save buffer 0 0K 1K127696K 16056 0 0
NK> 32,64,128
NK> vfscache 2604 419K 423K127696K 159971 0 0 64,256K
NK> BIO buffer 16 32K 505K127696K 18228 0 0 512,1K,2K
NK> pcb 174 17K 17K127696K 158237 0 0
NK> 16,32,64,128,2K
NK> soname 83 10K 11K127696K 21724454 0 0 16,32,64,128
NK> proc-args 53 2K 3K127696K 122092 0 0
NK> 16,32,64,128,256
NK> ptys 32 16K 16K127696K 32 0 0 512
NK> ttys 187 25K 50K127696K 23168 0 0 128,512
NK> kqueue 4 2K 5K127696K 47137 0 0 256,1K
NK> sigio 1 1K 1K127696K 2718 0 0 32
NK> file 521 33K 35K127696K 2102577 0 0 64
NK> file desc 75 20K 24K127696K 149811 0 0 256,512,1K
NK> shm 1 12K 12K127696K 1 0 0 16K
NK> dev_t 287 36K 36K127696K 287 0 0 128
NK> timecounter 10 2K 2K127696K 10 0 0 128
NK> kld 11 121K 126K127696K 34 0 0
NK> 16,32,128,1K,4K,32K,128K
NK> AR driver 1 1K 3K127696K 5 0 0 64,512,2K
NK> AD driver 2 2K 3K127696K 1500129 0 0 64,1K
NK> Gzip trees 0 0K 46K127696K 156997 0 0
NK> 32,64,128,256,512,1K,2K,4K,8K,32K
NK> ISOFS node 1097 138K 138K127696K 1097 0 0 128
NK> ISOFS mount 8 132K 132K127696K 8 0 0 512,128K
NK> sem 3 6K 6K127696K 3 0 0 1K,4K
NK> MD disk 2 2K 2K127696K 2 0 0 16,1K
NK> msg 4 25K 25K127696K 4 0 0 512,4K,16K
NK> rman 59 4K 4K127696K 461 0 0 16,64
NK> ioctlops 0 0K 2K127696K 1484 0 0 512,2K
NK> taskqueue 2 1K 1K127696K 2 0 0 32
NK> SWAP 2 413K 413K127696K 2 0 0 32,512K
NK> ATA generic 6 3K 3K127696K 6 0 0 16,512
NK> eventhandler 18 1K 1K127696K 18 0 0 32,64
NK> bus 359 31K 32K127696K 851 0 0
NK> 16,32,64,128,256,512,1K,2K,4K
NK> sysctl 0 0K 1K127696K 19399250 0 0 16,32,64
NK> uidinfo 3 1K 1K127696K 9367 0 0 32,128
NK> cred 14 2K 5K127696K 532789 0 0 128
NK> subproc 149 9K 12K127696K 301375 0 0 32,64,256
NK> proc 2 1K 1K127696K 2 0 0 512
NK> session 10 2K 3K127696K 12648 0 0 128
NK> pgrp 10 1K 1K127696K 12749 0 0 32
NK> ippool 1 1K 1K127696K 1 0 0 32
NK> pfestat 0 0K 1K127696K 4017935 0 0
NK> 16,32,64,128,512
NK> pic 3 1K 1K127696K 10 0 0 64,128
NK> iffclassstat 0 0K 40K127696K 19 0 0 4K
NK> lr 1 1K 1K127696K 1 0 0 512
NK> itable32 211 7K 7K127696K 211 0 0 32
NK> itable16 2236 537K 1656K127696K 10157 0 0 128,256
NK> ifstate242815 16288K 88751K127696K1711702529 0 0
NK> 16,32,64,128,256,512,1K
NK> pfe_ipc 0 0K 2K127696K4389469970 0 0
NK> 16,32,64,128,256,512,1K,2K
NK> mkey 453 8K 9K127696K 351154 0 0
NK> 16,32,64,128,1K
NK> ifaddr 223 14K 17K127696K 2476 0 0 64
NK> sysctl 0 0K 1K127696K 35 0 0 256
NK> rtable224823 14056K 27291K127696K348790726 0 0
NK> 16,32,64,128,256,512
NK> ifmaddr 80 2K 2K127696K 295 0 0 16,32
NK> ipfw 28 15K 19K127696K 988 0 0
NK> 16,32,64,128,256,512,1K,2K
NK> ifstat 240 99K 100K127696K 894 0 0 128,512,2K
NK> ifdevice 9 1K 1K127696K 9 0 0 32
NK> iflist 0 0K 1K127696K 27575 0 0 32
NK> rcache 4 8K 8K127696K 4 0 0 2K
NK> rnode214112 3663K 7071K127696K345160759 0 0 16,32,64,256
NK> metrics 21 3K 4K127696K 1417 0 0 128
NK> rtnexthop 17411 4320K 6650K127696K342565086 0 0
NK> 32,128,256,512
NK> iffamily 197 37K 38K127696K 608 0 0 128,256
NK>
NK> Memory Totals: In Use Free Requests
NK> 44866K 76567K 7198104864
NK>
NK> ITEM SIZE LIMIT USED FREE REQUESTS
NK>
NK> PIPE: 192, 0, 8, 77, 131628
NK> SWAPMETA: 160, 95772, 0, 0, 0
NK> unpcb: 160, 0, 114, 61, 74543
NK> ripcb: 192, 25323, 5, 37, 556
NK> syncache: 128, 15359, 0, 64, 88344
NK> tcpcb: 576, 25323, 36, 28, 126889
NK> udpcb: 192, 25323, 119, 51, 69246
NK> socket: 256, 25323, 369, 47, 338019
NK> KNOTE: 96, 0, 48, 78, 104503
NK> NFSNODE: 352, 0, 0, 0, 0
NK> NFSMOUNT: 544, 0, 0, 0, 0
NK> VNODE: 224, 0, 12104, 90, 12104
NK> NAMEI: 1024, 0, 0, 24, 5823314
NK> VMSPACE: 192, 0, 53, 75, 143957
NK> PROC: 448, 0, 70, 65, 143980
NK> DP fakepg: 64, 0, 0, 0, 0
NK> PV ENTRY: 28, 712366, 121953, 89112, 219769545
NK> MAP ENTRY: 48, 0, 1338, 915, 40350969
NK> KMAP ENTRY: 48, 35635, 210, 174, 173612
NK> MAP: 108, 0, 7, 3, 7
NK> VM OBJECT: 92, 0, 2688, 304, 2771913
NK>
NK> 2733497006 cpu context switches
NK> 1355685303 device interrupts
NK> 474779191 software interrupts
NK> 55982873 traps
NK> 54543811 system calls
NK> 17 kernel threads created
NK> 139746 fork() calls
NK> 4217 vfork() calls
NK> 0 rfork() calls
NK> 0 swap pager pageins
NK> 0 swap pager pages paged in
NK> 0 swap pager pageouts
NK> 0 swap pager pages paged out
NK> 381 vnode pager pageins
NK> 396 vnode pager pages paged in
NK> 25526 vnode pager pageouts
NK> 95451 vnode pager pages paged out
NK> 0 page daemon wakeups
NK> 0 pages examined by the page daemon
NK> 107 pages reactivated
NK> 13014131 copy-on-write faults
NK> 0 copy-on-write optimized faults
NK> 4163866 zero fill pages zeroed
NK> 3997676 zero fill pages prezeroed
NK> 2786 intransit blocking page faults
NK> 70463176 total VM faults taken
NK> 0 pages affected by kernel thread creation
NK> 115797006 pages affected by fork()
NK> 1620165 pages affected by vfork()
NK> 0 pages affected by rfork()
NK> 19948709 pages freed
NK> 0 pages freed by daemon
NK> 15065601 pages freed by exiting processes
NK> 45590 pages active
NK> 70954 pages inactive
NK> 0 pages in VM cache
NK> 45593 pages wired down
NK> 29407 pages free
NK> 4096 bytes per page
NK> 24839067 total name lookups
NK> cache hits (93% pos + 5% neg) system 0% per-directory
NK> deletions 0%, falsehits 0%, toolong 0%
NK> interrupt total rate
NK> ata0 irq14 2506559 0
NK> mux irq7 4093818368 600
NK> fxp1 irq10 15790 0
NK> sio0 irq4 2518 0
NK> clk irq0 681688963 99
NK> rtc irq8 872620401 127
NK> Total 5650652599 828
NK>
NK>
NK>
NK>
NK>
NK>
NK> --
NK> Accelerated IT Services GmbH
NK> Schubertstrasse 10 D-67251 Freinsheim
NK> nk at accelerated.de http://www.accelerated.de/
NK> Telefon: +49 6232 - 672568 Telefax: +49 6353 - 989039
NK> HRB: 60665 - Amtsgericht Ludwigshafen UstID: DE253684415
NK> Geschäftsführende Gesellschafter: Nicolaj Kamensek & Ole Krieger
NK>
NK>
NK>
More information about the juniper-nsp
mailing list