[j-nsp] syslog rsmon jtree problem
Cougar
cougar at random.ee
Mon Jan 4 06:57:40 EST 2010
On Tue, 12 May 2009, Juniper Iber-x wrote:
> We have restarted the SSB and routing processes, such as BGP sessions,
> is-is and so on, but the problem persists. We have decided to eliminate
> the logical router and thus freeing up memory for adding new prefixes.
Hello and HNY,
Did you find a solution for this? My router doing only local forwarding
and I see the same problem in M20 running 9.4R3.5 and using SSB IIv1.
cougar at R1> show chassis ssb
SSB status:
Slot 0 information:
State Master
Temperature 27 degrees C / 80 degrees F
CPU utilization 18 percent
Interrupt utilization 3 percent
Heap utilization 27 percent
Buffer utilization 60 percent
Total CPU DRAM 64 MB
Internet Processor II Version 1, Foundry IBM, Part number 9
Start time: 2010-01-02 09:31:38 UTC
Uptime: 1 day, 22 hours, 20 minutes, 20 seconds
…
[Jan 4 09:53:56.565 LOG: Warning] RSMON: Resource Category:jtree Instance:jtree0 Type:free-pages Available:0 is less than LWM limit:203, rsmon_syslog_limit()
[Jan 4 09:53:56.565 LOG: Warning] RSMON: Resource Category:jtree Instance:jtree0 Type:free-dwords Available:138 is less than LWM limit:26214, rsmon_syslog_limit()
[Jan 4 09:54:43.834 LOG: Err] RT: Failed prefix change IPv4:0 - 117.200.48/20, nh 262155, ifidx 0, nhifl 0, flag 0x10, cos 0
[Jan 4 09:54:43.834 LOG: Err] RT: Failed prefix change IPv4:0 - 117.200.48/20 (unknown prefix)
[Jan 4 09:54:56.566 LOG: Warning] RSMON: Resource Category:jtree Instance:jtree0 Type:free-pages Available:0 is less than LWM limit:203, rsmon_syslog_limit()
[Jan 4 09:54:56.566 LOG: Warning] RSMON: Resource Category:jtree Instance:jtree0 Type:free-dwords Available:391 is less than LWM limit:26214, rsmon_syslog_limit()
…
Prefixes I have seen in log (86.99.0/18, 41.236.72/22, 210.48.158/24, 41.236.72/22
etc) are not in this router routing table and even can't be. I really don't
know from where they can come in..
Here is what router shows about SSB
SSB0(R1 vty)# show rsmon
category instance type total lwm_limit hwm_limit free
-------- ----------- ------------ -------- --------- --------- --------
jtree jtree0 free-pages 4064 203 609 0
jtree jtree0 free-dwords 524288 26214 78643 544
SSB0(R1 vty)# show memory
ID Base Total(b) Free(b) Used(b) % Name
-- -------- --------- --------- --------- --- -----------
0 399d50 122053296 83805600 38247696 31 Kernel
1 97800000 8388608 4133036 4255572 50 Uncached
Looks like there are a lot of routes in SSB:
SSB0(R1 vty)# show jtree 0 summary
Protocol Routes Bytes Used
------------- ---------- ----------
IPv4 287650 4149760
IPv6 2146 37856
MPLS 5 72
Multi-service 1 16
Routes are all without next-hop information:
SSB0(R1 vty)# show jtree 0 ip
Route Type NH Index IFL Index Qn Dest
-------------------- -------------- ------------- --------- -- --------
Default Multiple(3) -
2.0/16 Multiple(3) -
2.1.0/21 Multiple(3) -
2.1.24/24 Multiple(3) -
3.51.92/23 Multiple(3) -
4/8 Multiple(3) -
4.0/9 Multiple(3) -
…
At the same time there are only some thousands routes in RIB..
cougar at R1> show pfe route summary
IPv4 Route Tables:
Index Routes Size(b)
-------- ---------- ----------
Default 2480 178402
1 9 635
2 6 425
MPLS Route Tables:
Index Routes Size(b)
-------- ---------- ----------
Default 4 281
IPv6 Route Tables:
Index Routes Size(b)
-------- ---------- ----------
Default 2537 186944
1 11 877
cougar at R1> show route summary
inet.0: 1115 destinations, 1251 routes (1113 active, 0 holddown, 10 hidden)
Direct: 164 routes, 162 active
Local: 165 routes, 165 active
BGP: 244 routes, 108 active
Static: 79 routes, 79 active
IS-IS: 594 routes, 594 active
Aggregate: 4 routes, 4 active
Access-internal: 1 routes, 1 active
inet.2: 594 destinations, 594 routes (594 active, 0 holddown, 0 hidden)
IS-IS: 594 routes, 594 active
__juniper_private1__.inet.0: 4 destinations, 4 routes (3 active, 0 holddown, 1 hidden)
Direct: 2 routes, 2 active
Local: 2 routes, 1 active
__juniper_private2__.inet.0: 1 destinations, 1 routes (0 active, 0 holddown, 1 hidden)
Direct: 1 routes, 0 active
iso.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
Direct: 1 routes, 1 active
mpls.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
MPLS: 3 routes, 3 active
inet6.0: 2380 destinations, 2415 routes (2380 active, 0 holddown, 4 hidden)
Direct: 44 routes, 20 active
Local: 42 routes, 42 active
BGP: 2248 routes, 2237 active
Static: 14 routes, 14 active
IS-IS: 66 routes, 66 active
Aggregate: 1 routes, 1 active
__juniper_private1__.inet6.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
Direct: 2 routes, 2 active
Local: 2 routes, 2 active
It looks like full table leaked to this router in some moment (by config
it cant happen) and PFE is now out of sync and didn't remove prefixes..
Doing debug lookup I also get nexthop errors for these prefixes like this:
SSB0(R1 vty)# show jtree 0 debug ip lookup 117.200.48/20
Location Value Type Bit Next
---------- -------- -------- --- --------
117 stack 00010200 attached 0 40
117 search 05017e08 4-way 8 1405e
root(seg:0) 00010200 attached 0 40
40 0006c801 2-way 1 1b2
1b3 00363e02 4-way 2 d8e
d91 00a96e04 4-way 4 2a5a
2a5b 00cdb606 4-way 6 336c
336d 05017e08 4-way 8 1405e
14061 01d5c00a 2-way 10 7570
7570 1f34d80b 2-way 11 7cd36
7cd36 07be3e0c 4-way 12 1ef8e
1ef90 03ec260e 4-way 14 fb08
fb08 1f334610 4-way 16 7ccd0
7ccd0 2b188812 2-way 18 ac622
ac623 1f339414 child 20 7cce4
7cce5 75c82000 prefix
*** unsupported next hop 000d5903 at location 42
Is there any way to clear FIB and initiate new RIB->FIB update without
restarting SSB (which makes too long break)?
--
Cougar
More information about the juniper-nsp
mailing list