[j-nsp] Juniper M20 memory problem

Josef Buchsteiner josefb at juniper.net
Mon Jun 11 03:10:56 EDT 2007


Nicolaj,


        I  was more trying to understand why the kernel is complaining
        of

        "Nexthop   index   allocation  failed:  regular  index  space
        exhausted"

        In   this   version  the  regular  index  space  from  kernels
        perspective  is  256k.  Before  you  reach  such limit you are
        running  out  of resources either on the FPC due to Descriptor
        space  shortage  or due to DRAM Memory shortage on the S-Board
        so this should never we the bottleneck in the version you use.
        Anyway, I did not got to an answer here and maybe I would need
        to  complete  message  log  and or remote login. The number of
        rtnexthops  is  17411  entries as you can see from the virtual
        memory point of view.


        From the data below we can see that the mas next-hop entries
        was 8106 unicast next-hops on both FPCs. You had double of
        those entries since you had an aggregate with 2 member links
        and you can see already the number of next-hops and resources
        depends on the number of memberlinks of an aggregate. Also
        the  L2  Descriptor  space  was  with  32532 entries below the
        limit. Any increase you would need to have an Enhanced FPC.


        
        
 SSB1(re0 vty)# show nhdb zones
 Chip  Start   Size   Rsvd   Used/Hi Water/Total  Size  Name
 ----  -----  -----  -----  --------------------  ----  ----
     0  20000  01000  00000               0/0/512     8  Multicast RTP
     0  21000  02400  00000              0/0/9216     1  Multicast Lists
     0  23400  04400  00006       7202/8106/17408     1  Next-Hop Entries
     0  27800  18800  0004b     28928/32532/50176     2  L2 Descriptors
     0  40400  001ff  00000                3/3/63     8  L2 Programs
     1  20000  01000  00000               0/0/512     8  Multicast RTP
     1  21000  02400  00000              0/0/9216     1  Multicast Lists
     1  23400  04400  00006          0/8106/17408     1  Next-Hop Entries
     1  27800  18800  0004b       120/32532/50176     2  L2 Descriptors
     1  40400  001ff  00000                2/3/63     8  L2 Programs        


       If  you have arp entries only with vlans and without vlans here
       are  quick  summary  on how far you can go depending on the FPC
       Type. Those numbers are per FPC and not per system and the DRAM
       memory of the S-Board is the other part which could limit the
       numbers.

       
    arp entries | ether  | with vlan
    ------------+--------+------------
       FPC      |  16750 | 13060
                |        |
       E-FPC    |  54272 | 40722
                |        |
       E2-FPC   |  61183 | 61183
       M10i/M7i |        |
 


       Since you were not running into those merits you indicated that
       you  were  running  out  of memory resources on the S-Board and
       your  utilization  is already about 76%. As a guideline usually
       you  should  look  at  the  details  between  80-90%  where the
       resources   are   going   to  better manage this for future and
       plan  the  upgrade  or changes. Any transient change might need
       some high peeks of resources and you should have some room left
       to  make  such  kind  of operations. Once you are at 90% then I
       usually suggest for an immediate actions in short time frames.


       In  order  to illustrate how an aggregate is build and how much
       memory we need here an example
       


       Route  -> Aggregate next-hop ---> unicast next-hop
                           |
                           +-----------> unicast next-hop

                           
      route entry:
      ============
      ipv4 route is 68byte + 4byte(ipv4)  = 72byte
      (If you have uRPF enabled you can add 52byte to each route)


      unicast nexthop:
      ================
      each  of  them  is  348  byte which includes the max possible
      L2 header
       

      Aggregate next-hop:
      ===================
      92 + (x * 40) where x is the number of links in an aggregate
      which is in your case 172byte.


      So  in  your  case  with  8k  of  arp  entries and you create an
      aggregate of 2 member links you need to following more resources
      on memory space. You double the unicast entries and you add
      an additional aggregate next-hops for each route.

      172byte * 8106 = 1,39MB
      384byte * 8106 = 3.11MB

      So in total about 4,5MB more.

      
      If you have the enhanced SSB-E with 256M then this will never be
      the restriction and you would need to  upgrade the FPCs to E-FPC
      and  then  you  can  have  54k  entries on each FPCs and getting
      really high numbers of arp entries. Of course of you have then
      100k  or  150k  on arp entries you would need to think on how to
      deal  with  arp-refresh  and so on and the regular resources you
      have on the RE. e.g. RE3.0 is recommended.


      hope this helps a bit.

      Josef
      

      

       


Sunday, June 10, 2007, 3:24:44 PM, you wrote:
NK>    
NK>    
NK> Josef Buchsteiner wrote:
NK>  
NK>  Hi Josef,
NK>  
 >> show version
 >> show system virtual-memory
NK>  
NK>  the output is below. But meanwhile of course I had to fix the problem
NK>  and therefore moved some Gateway IPs to a different router to decrease
NK>  the arp count from 7200 to 5611 currently. I had to reboot the SSB-E to
NK>  free the memory but at least it is working now.
NK>  
NK>  nico at re0> show version
NK>  Hostname: re0
NK>  Model: m20
NK>  JUNOS Base OS boot [7.6R1.10]
NK>  JUNOS Base OS Software Suite [7.6R1.10]
NK>  JUNOS Kernel Software Suite [7.6R1.10]
NK>  JUNOS Packet Forwarding Engine Support (M20/M40) [7.6R1.10]
NK>  JUNOS Routing Software Suite [7.6R1.10]
NK>  JUNOS Online Documentation [7.6R1.10]
NK>  JUNOS Crypto Software Suite [7.6R1.10]
NK>  
NK>  
NK>  
NK>  nico at re0> show system virtual-memory
NK>  Memory statistics by bucket size
NK>  Size   In Use   Free   Requests  HighWater  Couldfree
NK>     16   196393 183767 2355256898    1280    1140495
NK>     32    30503 249817 1418360018     640     298805
NK>     64   460785 515407 1405666746     320    1857754
NK>    128    20788 244748  686148930     160     944141
NK>    256    21129  11255  992196792      80    1145025
NK>    512      113     15  340207290      40          0
NK>     1K       81      7     105736      20         52
NK>     2K      195     87      17036      10       3309
NK>     4K       17     10      55228       5       1744
NK>     8K       73      5      87868       5        669
NK>    16K        8      0         15       5          0
NK>    32K       12      0       2281       5          0
NK>    64K       18      0         18       5          0
NK>  128K        5      0          5       5          0
NK>  256K        2      0          2       5          0
NK>  512K        1      0          1       5          0
NK>  
NK>  Memory usage type by bucket size
NK>  Size  Type(s)
NK>     16  uc_devlist, nexusdev, iftable, temp, devbuf, atexit, COS, BPF,
NK>             DEVFS mount, DEVFS node, vnodes, mount, pcb, soname,
NK>  proc-args, kld,
NK>             MD disk, rman, ATA generic, bus, sysctl, pfestat, ifstate,
NK>  pfe_ipc,
NK>             mkey, rtable, ifmaddr, ipfw, rnode
NK>     32  atkbddev, dirrem, mkdir, diradd, freefile, freefrag, indirdep,
NK>             bmsafemap, newblk, temp, devbuf, tseg_qent, COS, vnodes,
NK>             cluster_save buffer, pcb, soname, proc-args, sigio, kld, Gzip
NK>  trees,
NK>             taskqueue, SWAP, eventhandler, bus, sysctl, uidinfo, subproc,
NK>  pgrp,
NK>             ippool, pfestat, itable32, ifstate, pfe_ipc, mkey, rtable,
NK>  ifmaddr,
NK>             ipfw, ifdevice, iflist, rnode, rtnexthop
NK>     64  isadev, ip6ndp, MFS node, allocindir, allocdirect, pagedep, temp,
NK>             devbuf, lockf, COS, NULLFS hash, DEVFS name, vnodes,
NK>             cluster_save buffer, vfscache, pcb, soname, proc-args, file,
NK>             AR driver, AD driver, Gzip trees, rman, eventhandler, bus,
NK>  sysctl,
NK>             subproc, pfestat, pic, ifstate, pfe_ipc, mkey, ifaddr,
NK>  rtable, ipfw,
NK>             rnode
NK>    128  ZONE, freeblks, inodedep, temp, devbuf, zombie, COS, BPF, DEVFS node,
NK>             vnodes, mount, cluster_save buffer, pcb, soname, proc-args, ttys,
NK>             dev_t, timecounter, kld, Gzip trees, ISOFS node, bus,
NK>  uidinfo, cred,
NK>             session, pfestat, pic, itable16, ifstate, pfe_ipc, mkey, rtable,
NK>             ipfw, ifstat, metrics, rtnexthop, iffamily
NK>    256  iflogical, iftable, MFS node, FFS node, newblk, temp, devbuf,
NK>             NFS daemon, vnodes, proc-args, kqueue, file desc, Gzip trees,
NK>  bus,
NK>             subproc, itable16, ifstate, pfe_ipc, sysctl, rtable, ipfw, rnode,
NK>             rtnexthop, iffamily
NK>    512  UFS mount, temp, devbuf, mount, BIO buffer, ptys, ttys, file desc,
NK>             AR driver, Gzip trees, ISOFS mount, msg, ioctlops, ATA
NK>  generic, bus,
NK>             proc, pfestat, lr, ifstate, pfe_ipc, rtable, ipfw, ifstat,
NK>  rtnexthop
NK>     1K  iftable, temp, devbuf, NQNFS Lease, COS, BIO buffer, kqueue,
NK>             file desc, kld, AD driver, Gzip trees, sem, MD disk, bus,
NK>  ifstate,
NK>             pfe_ipc, mkey, ipfw
NK>     2K  uc_devlist, iflogical, UFS mount, temp, devbuf, BIO buffer, pcb,
NK>             AR driver, Gzip trees, ioctlops, bus, pfe_ipc, ipfw, ifstat,
NK>  rcache
NK>     4K  memdesc, iftable, UFS mount, temp, devbuf, kld, Gzip trees, sem, msg,
NK>             bus, iffclassstat
NK>     8K  iflogical, indirdep, temp, devbuf, syncache, Gzip trees
NK>    16K  UFS mount, temp, devbuf, shm, msg
NK>    32K  pagedep, devbuf, BPF, kld, Gzip trees
NK>    64K  VM pgdata, devbuf, MSDOSFS mount
NK>  128K  UFS ihash, inodedep, NFS hash, kld, ISOFS mount
NK>  256K  mbuf, vfscache
NK>  512K  SWAP
NK>  
NK>  Memory statistics by type                          Type  Kern
NK>           Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
NK>          isadev    13     1K      1K127696K       13    0     0  64
NK>        atkbddev     2     1K      1K127696K        2    0     0  32
NK>      uc_devlist    24     3K      3K127696K       24    0     0  16,2K
NK>        nexusdev     3     1K      1K127696K        3    0     0  16
NK>         memdesc     1     4K      4K127696K        1    0     0  4K
NK>            mbuf     1   152K    152K127696K        1    0     0  256K
NK>       iflogical   159   558K    576K127696K      543    0     0  256,2K,8K
NK>         iftable    16     9K      9K127696K       28    0     0  16,256,1K,4K
NK>          ip6ndp     0     0K      1K127696K        2    0     0  64
NK>            ZONE    15     2K      2K127696K       15    0     0  128
NK>       VM pgdata     1    64K     64K127696K        1    0     0  64K
NK>       UFS mount    15    41K     41K127696K       15    0     0
NK>  512,2K,4K,16K
NK>       UFS ihash     1   128K    128K127696K        1    0     0  128K
NK>        MFS node   261    65K     72K127696K    11312    0     0  64,256
NK>        FFS node  1212   303K    304K127696K    10321    0     0  256
NK>          dirrem     0     0K      4K127696K    41400    0     0  32
NK>           mkdir     0     0K      1K127696K        8    0     0  32
NK>          diradd     0     0K      2K127696K    41433    0     0  32
NK>        freefile     0     0K      1K127696K     7302    0     0  32
NK>        freeblks     0     0K      1K127696K     8682    0     0  128
NK>        freefrag     0     0K      3K127696K   135820    0     0  32
NK>      allocindir     0     0K    248K127696K   406886    0     0  64
NK>        indirdep     0     0K     33K127696K   166778    0     0  32,8K
NK>     allocdirect     0     0K      4K127696K   232268    0     0  64
NK>       bmsafemap     0     0K      1K127696K   146306    0     0  32
NK>          newblk     1     1K      1K127696K   639155    0     0  32,256
NK>        inodedep     1   128K    137K127696K    53025    0     0  128,128K
NK>         pagedep     1    32K     33K127696K     5813    0     0  64,32K
NK>            temp  1337    94K     99K127696K   337591    0     0
NK>  16,32,64,128,256,512,1K,2K,4K,8K,16K
NK>          devbuf 16825  2350K   2671K127696K  1042931    0     0
NK>  16,32,64,128,256,512,1K,2K,4K,8K,16K,32K,64K
NK>           lockf    42     3K      3K127696K  6112563    0     0  64
NK>          atexit     1     1K      1K127696K        1    0     0  16
NK>          zombie     0     0K      1K127696K   143910    0     0  128
NK>        NFS hash     1   128K    128K127696K        1    0     0  128K
NK>     NQNFS Lease     1     1K      1K127696K        1    0     0  1K
NK>      NFS daemon     1     1K      1K127696K        1    0     0  256
NK>        syncache     1     8K      8K127696K        1    0     0  8K
NK>       tseg_qent     0     0K      1K127696K     1452    0     0  32
NK>             COS    87    63K     65K127696K      349    0     0
NK>  16,32,64,128,1K
NK>             BPF   403     7K     69K127696K      524    0     0  16,128,32K
NK>  MSDOSFS mount     1    64K     64K127696K        1    0     0  64K
NK>     NULLFS hash     1     1K      1K127696K        1    0     0  64
NK>     DEVFS mount     2     1K      1K127696K        2    0     0  16
NK>      DEVFS name   493    31K     31K127696K      493    0     0  64
NK>      DEVFS node   479    59K     59K127696K      517    0     0  16,128
NK>          vnodes    28     7K      7K127696K      429    0     0
NK>  16,32,64,128,256
NK>           mount    15     8K      8K127696K       17    0     0  16,128,512
NK>  cluster_save buffer     0     0K      1K127696K    16056    0     0
NK>  32,64,128
NK>        vfscache  2604   419K    423K127696K   159971    0     0  64,256K
NK>      BIO buffer    16    32K    505K127696K    18228    0     0  512,1K,2K
NK>             pcb   174    17K     17K127696K   158237    0     0
NK>  16,32,64,128,2K
NK>          soname    83    10K     11K127696K 21724454    0     0  16,32,64,128
NK>       proc-args    53     2K      3K127696K   122092    0     0
NK>  16,32,64,128,256
NK>            ptys    32    16K     16K127696K       32    0     0  512
NK>            ttys   187    25K     50K127696K    23168    0     0  128,512
NK>          kqueue     4     2K      5K127696K    47137    0     0  256,1K
NK>           sigio     1     1K      1K127696K     2718    0     0  32
NK>            file   521    33K     35K127696K  2102577    0     0  64
NK>       file desc    75    20K     24K127696K   149811    0     0  256,512,1K
NK>             shm     1    12K     12K127696K        1    0     0  16K
NK>           dev_t   287    36K     36K127696K      287    0     0  128
NK>     timecounter    10     2K      2K127696K       10    0     0  128
NK>             kld    11   121K    126K127696K       34    0     0
NK>  16,32,128,1K,4K,32K,128K
NK>       AR driver     1     1K      3K127696K        5    0     0  64,512,2K
NK>       AD driver     2     2K      3K127696K  1500129    0     0  64,1K
NK>      Gzip trees     0     0K     46K127696K   156997    0     0
NK>  32,64,128,256,512,1K,2K,4K,8K,32K
NK>      ISOFS node  1097   138K    138K127696K     1097    0     0  128
NK>     ISOFS mount     8   132K    132K127696K        8    0     0  512,128K
NK>             sem     3     6K      6K127696K        3    0     0  1K,4K
NK>         MD disk     2     2K      2K127696K        2    0     0  16,1K
NK>             msg     4    25K     25K127696K        4    0     0  512,4K,16K
NK>            rman    59     4K      4K127696K      461    0     0  16,64
NK>        ioctlops     0     0K      2K127696K     1484    0     0  512,2K
NK>       taskqueue     2     1K      1K127696K        2    0     0  32
NK>            SWAP     2   413K    413K127696K        2    0     0  32,512K
NK>     ATA generic     6     3K      3K127696K        6    0     0  16,512
NK>    eventhandler    18     1K      1K127696K       18    0     0  32,64
NK>             bus   359    31K     32K127696K      851    0     0
NK>  16,32,64,128,256,512,1K,2K,4K
NK>          sysctl     0     0K      1K127696K 19399250    0     0  16,32,64
NK>         uidinfo     3     1K      1K127696K     9367    0     0  32,128
NK>            cred    14     2K      5K127696K   532789    0     0  128
NK>         subproc   149     9K     12K127696K   301375    0     0  32,64,256
NK>            proc     2     1K      1K127696K        2    0     0  512
NK>         session    10     2K      3K127696K    12648    0     0  128
NK>            pgrp    10     1K      1K127696K    12749    0     0  32
NK>          ippool     1     1K      1K127696K        1    0     0  32
NK>         pfestat     0     0K      1K127696K  4017935    0     0
NK>  16,32,64,128,512
NK>             pic     3     1K      1K127696K       10    0     0  64,128
NK>    iffclassstat     0     0K     40K127696K       19    0     0  4K
NK>              lr     1     1K      1K127696K        1    0     0  512
NK>        itable32   211     7K      7K127696K      211    0     0  32
NK>        itable16  2236   537K   1656K127696K    10157    0     0  128,256
NK>         ifstate242815 16288K  88751K127696K1711702529    0     0
NK>  16,32,64,128,256,512,1K
NK>         pfe_ipc     0     0K      2K127696K4389469970    0     0
NK>  16,32,64,128,256,512,1K,2K
NK>            mkey   453     8K      9K127696K   351154    0     0
NK>  16,32,64,128,1K
NK>          ifaddr   223    14K     17K127696K     2476    0     0  64
NK>          sysctl     0     0K      1K127696K       35    0     0  256
NK>          rtable224823 14056K  27291K127696K348790726    0     0
NK>  16,32,64,128,256,512
NK>         ifmaddr    80     2K      2K127696K      295    0     0  16,32
NK>            ipfw    28    15K     19K127696K      988    0     0
NK>  16,32,64,128,256,512,1K,2K
NK>          ifstat   240    99K    100K127696K      894    0     0  128,512,2K
NK>        ifdevice     9     1K      1K127696K        9    0     0  32
NK>          iflist     0     0K      1K127696K    27575    0     0  32
NK>          rcache     4     8K      8K127696K        4    0     0  2K
NK>           rnode214112  3663K   7071K127696K345160759    0     0  16,32,64,256
NK>         metrics    21     3K      4K127696K     1417    0     0  128
NK>       rtnexthop 17411  4320K   6650K127696K342565086    0     0
NK>  32,128,256,512
NK>        iffamily   197    37K     38K127696K      608    0     0  128,256
NK>  
NK>  Memory Totals:  In Use       Free    Requests
NK>                   44866K     76567K    7198104864
NK>  
NK>  ITEM            SIZE     LIMIT    USED    FREE  REQUESTS
NK>  
NK>  PIPE:            192,        0,      8,     77,   131628
NK>  SWAPMETA:        160,    95772,      0,      0,        0
NK>  unpcb:           160,        0,    114,     61,    74543
NK>  ripcb:           192,    25323,      5,     37,      556
NK>  syncache:        128,    15359,      0,     64,    88344
NK>  tcpcb:           576,    25323,     36,     28,   126889
NK>  udpcb:           192,    25323,    119,     51,    69246
NK>  socket:          256,    25323,    369,     47,   338019
NK>  KNOTE:            96,        0,     48,     78,   104503
NK>  NFSNODE:         352,        0,      0,      0,        0
NK>  NFSMOUNT:        544,        0,      0,      0,        0
NK>  VNODE:           224,        0,  12104,     90,    12104
NK>  NAMEI:          1024,        0,      0,     24,  5823314
NK>  VMSPACE:         192,        0,     53,     75,   143957
NK>  PROC:            448,        0,     70,     65,   143980
NK>  DP fakepg:        64,        0,      0,      0,        0
NK>  PV ENTRY:         28,   712366, 121953,  89112, 219769545
NK>  MAP ENTRY:        48,        0,   1338,    915, 40350969
NK>  KMAP ENTRY:       48,    35635,    210,    174,   173612
NK>  MAP:             108,        0,      7,      3,        7
NK>  VM OBJECT:        92,        0,   2688,    304,  2771913
NK>  
NK>  2733497006 cpu context switches
NK>  1355685303 device interrupts
NK>  474779191 software interrupts
NK>    55982873 traps
NK>    54543811 system calls
NK>          17 kernel threads created
NK>      139746  fork() calls
NK>        4217 vfork() calls
NK>           0 rfork() calls
NK>           0 swap pager pageins
NK>           0 swap pager pages paged in
NK>           0 swap pager pageouts
NK>           0 swap pager pages paged out
NK>         381 vnode pager pageins
NK>         396 vnode pager pages paged in
NK>       25526 vnode pager pageouts
NK>       95451 vnode pager pages paged out
NK>           0 page daemon wakeups
NK>           0 pages examined by the page daemon
NK>         107 pages reactivated
NK>    13014131 copy-on-write faults
NK>           0 copy-on-write optimized faults
NK>     4163866 zero fill pages zeroed
NK>     3997676 zero fill pages prezeroed
NK>        2786 intransit blocking page faults
NK>    70463176 total VM faults taken
NK>           0 pages affected by kernel thread creation
NK>  115797006 pages affected by  fork()
NK>     1620165 pages affected by vfork()
NK>           0 pages affected by rfork()
NK>    19948709 pages freed
NK>           0 pages freed by daemon
NK>    15065601 pages freed by exiting processes
NK>       45590 pages active
NK>       70954 pages inactive
NK>           0 pages in VM cache
NK>       45593 pages wired down
NK>       29407 pages free
NK>        4096 bytes per page
NK>    24839067 total name lookups
NK>             cache hits (93% pos + 5% neg) system 0% per-directory
NK>             deletions 0%, falsehits 0%, toolong 0%
NK>  interrupt                   total       rate
NK>  ata0 irq14                2506559          0
NK>  mux irq7               4093818368        600
NK>  fxp1 irq10                  15790          0
NK>  sio0 irq4                    2518          0
NK>  clk irq0                681688963         99
NK>  rtc irq8                872620401        127
NK>  Total                  5650652599        828
NK>  
NK>  
NK>  
NK>  
NK>  
NK>  
NK>  --
NK>  Accelerated IT Services GmbH
NK>  Schubertstrasse 10                        D-67251 Freinsheim
NK>  nk at accelerated.de                         http://www.accelerated.de/
NK>  Telefon: +49 6232 - 672568                Telefax: +49 6353 - 989039
NK>  HRB: 60665 - Amtsgericht Ludwigshafen     UstID: DE253684415
NK>  Geschäftsführende Gesellschafter:         Nicolaj Kamensek & Ole Krieger
NK>   
NK>   
NK>    

 


More information about the juniper-nsp mailing list