[c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

Erik Sundberg ESundberg at nitelusa.com
Thu Jun 2 15:41:16 EDT 2016


We have been using ASR920's for a couple months now.

I have an outstanding Memory leak issue in asr920-universalk9_npe.03.16.01a.S.155-3.S1a-ext.bin

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy87268

The work around doesn't work for me, I just tried it. TAC gave me the following. We have to reboot the ASR920 to lower the memory back down.

The fix for CSCuy87268 has been committed and would be available in the following releases:

XE 3.16.4S/15.5(3)S4 - planned for September 2016.
XE 3.18.S1/15.6(2)S1 - planned for mid June 2016.



Issue RP0 memory usage at 98% and growing, status is in the warning state. I had to reboot the device to get the memory back down, however it still grows.


Model: ASR-920-24SZ-M

ASR920 - #1
98% used memory and uptime is 24 weeks.
ASR920#sh platform software status control-processor bri
Load Average
 Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   0.04   0.07   0.04

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Warning  3438048  3361672 (98%)    76376 ( 2%)   3345388 (97%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
  RP0    0  11.93   7.22   0.00  80.64   0.00   0.20   0.00
         1   9.60   8.30   0.00  81.78   0.00   0.30   0.00


ASR920 - #2 Before Reboot (Uptime around 10 Weeks)

ASR920#sh platform software status control-processor bri
Load Average
  Slot  Status  1-Min  5-Min 15-Min
   RP0 Healthy   0.00   0.00   0.00

Memory (kB)
  Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Warning  3438048  3360436 (98%)    77612 ( 2%)   3341328 (97%)

CPU Utilization
  Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
   RP0    0   6.60   3.70   0.00  89.38   0.00   0.30   0.00
        1   1.40   0.80   0.00  97.80   0.00   0.00   0.00


ASR920 - #2 Post Reboot

EAR1.ATL1#sh platform software status control-processor bri
Load Average
 Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   0.00   0.02   0.00

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Healthy  3438048  1855832 (54%)  1582216 (46%)   1512524 (44%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
  RP0    0  10.68  10.88   0.00  78.02   0.00   0.39   0.00
         1   6.29   6.49   0.00  86.91   0.00   0.29   0.00




The memory leaks in the process SPA_XCVR_OIR and  enqueue_oir_msg  from what tac has told me.

EAR1.ATL1#  show platform software memory iomd 0/0 brief
  module                  allocated     requested     allocs        frees
  ------------------------------------------------------------------------------
  DEVOBJ                  38496         37792         88            0
  IOMd intr               1392          1104          36            0
  Summary                 1198313647    679413327     496656324     431793784
  appsess_ctx             1736          1728          1             0
  appsess_timer           56            40            3             1
  bsess_hdl               40            32            1             0
  cdh-shim                7260          5940          165           0
  cdllib                  1668          1660          7             6
  chunk                   5573          5517          7             0
  enqueue_oir_msg         506580480     253290240     49236333      17575053
  env_wheel               20108         20100         1             0
  eventutil               310794        307834        386           16
  fpd_sb                  208           200           1             0
  fpd_upg                 5636          5548          11            0
  geim_esb                5040          4816          28            0
  geim_hwidb              16352         16128         28            0
  geim_instance           16800         16576         28            0
  geim_spa_instance       13440         13216         28            0
  geim_spa_plugin         764           756           2             1
  ipc_shim_pak            0             0             17057467      17057467
  null_spa_plugin         104           96            1             0
  oir_create              80            72            1             0
  oir_enqueue_event       0             0             1             1
  oir_processing          24            16            1             0
  queue                   240           200           5             0
  spa_bay_array           128           96            4             0
  spa_bay_create          120           112           1             0
  spa_env_enq             0             0             7             7
  spa_env_subsys_init     512           384           16            0
  spa_oir_psm             264           240           28            25
  spa_plugin              672           480           24            0
  spa_tdl_alloc           0             0             343085676     343085676
  spa_xcvr_oir            691264468     425661956     50775367      17575053
  tdl                     0             0             36500281      36500281
  tdl_hdl                 640           472           21            0
  tdl_ping_stats          5096          5024          9             0
  trccfg                  256           248           5             4
  uipeer                  9288          9264          15            12
  unknown                 0             0             181           181
  unknown                 120           112           1             0
  watched_bitfield        36            28            1             0
  xcvr                    5084          4852          29            0
  xcvr_oir_timer          672           448           28            0



-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of CiscoNSP List
Sent: Thursday, May 26, 2016 4:50 AM
To: Mark Tinka; James Jun
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

Cheers Guys - Have 03.16.02aS on the ones ready for deployment, so fingers crossed, it remains a stable release :)

Thanks for the feedback.

________________________________________
From: Mark Tinka <mark.tinka at seacom.mu>
Sent: Thursday, 26 May 2016 3:35 PM
To: James Jun; CiscoNSP List
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

On 26/May/16 06:23, James Jun wrote:

> I believe there is an mpls label-range bug that Mark noted (CSCuy29638) which is outstanding, but that can be worked around by staying out of high label range.

Actually, Cisco normally allocate low-range labels. So this is fine if the box adjacent to the ASR920 is a Cisco.

Juniper tend to allocate labels in the 300,000 - 400,000+ range. This is not an issue if the ASR920 is not adjacent to the Juniper. In our case, all situations where we hit this bug is when the ASR920 is adjacent to a Juniper. We are not in a position to change the topology to avoid the adjacency between the ASR920 and a Juniper.

Unfortunately, it is not possible to set a label allocation range in Junos today, the same way you can on a Cisco. That capability is only slated for Junos 16.

For us, the workaround is a test image that includes the fix that the BU built specially for us.

But not to worry, the BU say the fixed image is still on schedule for release 7th June, this year :-).

>
>
> Thankfully, this bug is also fixed on 03.16.02aS.  920s upgraded to this release appear to be fine so far.

Save for my MPLS label range issue, this release is very stable.

Mark.
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.


More information about the cisco-nsp mailing list