[c-nsp] Router down - What is "Net Background"?

Fri Apr 4 16:01:54 EDT 2008

The problem turned out to be a bad NM-1A-OC3SMI.  I scrounged up a spare 
and replaced it and resolved the issue.  The bouncing circuit was 
definitely the problem.  Even with all the sub-ints admin down the 
bouncing of the OC3 itself was enough to make the 3660 waiver.  I can 
only imagine what it would have been like if I hadn't had the sub-ints 
down.  Actually, I guess I don't have to try and imagine it; it happened 
this AM.

The router did actually crash.  3 times this AM.  I have 3 crash logs to 
work with.  I'm going to use them to RMA the NM with TAC.

All is well right now.  I'm surprised at how painful an OC3 state change 
can be.  I don't recall ever having that kind of drain on the resources 
when working with them before.  It's something to remember though.

Thanks for all the info,
  Justin

e ninja wrote:
> Justin,
> 
> What do you mean by "die"? A crash is not the same thing as a hang. Did 
> your 3660 crash or hang? If the former, IOS should dump a crashinfo file 
> - if you have one, lets take a look at it. If the latter, take a look at;
> 
> http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a0080106fd7.shtml
> 
> /eninja
> 
> 
> 
> 
> On Fri, Apr 4, 2008 at 7:42 AM, Justin Shore <justin at justinshore.com 
> <mailto:justin at justinshore.com>> wrote:
> 
>     I just had a 3660 terminating about 1500 PVCs for DSL die this AM.  The
>     3660 is running 12.3(22) w/ 256MB of RAM.  I'm at the remote POP now,
>     consoled in and looking at the box.  The console hangs for 45-60s before
>     I get a chance to run a quick command or two.  Then it goes back to the
>     hung state about 5 seconds later.  The box keeps dropping my 2 OC3s and
>     bouncing my IS-IS processes.  Syslog didn't appear to capture anything
>     interesting leading up to this crash.  I had the box power cycled while
>     I was driving down here.
> 
>     Overall the CPU is low and neither CPU nor memory were being taxed over
>     the past number of months.  The Net Background process is using a fair
>     bit of the processor.  Actually, that's an understatement.  At times
>     it's only using 9% or so.  At other times it's up to 97% as was the case
>     here:
> 
>     3660-2.brd#sh proc cpu sort 5m
>     CPU utilization for five seconds: 1%/0%; one minute: 17%; five
>     minutes: 19%
>      PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>       28     4715940       611    7731049 97.09% 13.74%  9.23%   0 Net
>     Background
>       31      238608       865     275847  2.65%  8.58%  8.48%   0
>     Per-Second Jobs
>        5        8336       684      12187  0.00%  0.56%  0.68%   0 Check
>     heaps
>       60        7764      7776        998  0.01%  0.27%  0.43%   0 IP Input
> 
>       18        3856      8741        441  0.00%  0.42%  0.33%   0 ARP
>     Input
>      167        2556      3046        839  0.02%  0.12%  0.13%   0 ISIS Upd
> 
>      140        2732        15     182133  0.00%  0.00%  0.12%   0 Key Proc
> 
>      179        1680      1676       1002  0.00%  0.13%  0.11%   0 OSPF
>     Router
>       47        1912       152      12578  0.00%  0.12%  0.11%   0
>     Per-minute Jobs
>       82        1792       597       3001  0.01%  0.15%  0.11%   0 IP
>     Background
>        3        1828       238       7680  0.00%  0.11%  0.09%   0 OSPF
>     Hello
>      100         644      1217        529  0.00%  0.13%  0.08%   0 DHCPD
>     Receive
>       51        1700       376       4521  0.00%  0.04%  0.05%   0 ATM
>     Periodic
>       83         700       417       1678  0.00%  0.01%  0.02%   0 IP RIB
>     Update
>      108         500       871        574  0.00%  0.06%  0.02%   0 CEF
>     process
>      124         172       471        365  0.00%  0.04%  0.02%   0 Exec
> 
>      166         620      1908        324  0.00%  0.03%  0.02%   0 ISIS Adj
> 
>      106         420       284       1478  0.00%  0.03%  0.00%   0 Adj
>     Manager
>        6         436      1406        310  0.00%  0.02%  0.00%   0 Pool
>     Manager
>       91         188       556        338  0.00%  0.01%  0.00%   0 ILMI
>     Timer Proce
>      163          92      1795         51  0.00%  0.00%  0.00%   0 CLNS
>     Input
> 
> 
>     Per-Second Jobs is also using quite a bit of CPU.  The log has CPUHOG
>     entries in it pointing at the Net Background process too:
> 
>     000428: .Apr  4 08:36:31 CDT: %SYS-3-CPUHOG: Task ran for 50332 msec
>     (10/0), process = Net Background, PC = 60439FFC.
>     -Traceback= 6043A004
> 
>     What does the Net Background process do?  I haven't been able to find
>     any answers on Cisco.com.
> 
> 
>     We're in the middle of SmartNet renewals and negotiations and
>     unfortunately this box's contract has expired.  I can have our AM bless
>     a TAC case for us if need be, assuming I can locate him; I think he's in
>     training this week back East.
> 
>     Any ideas?  I'm going to go back and scrutinize the log for anything
>     useful.  As of right now the box is intermittently pingable,
>     corresponding directly to the brief instances that I can use the
>     console.  Customer packets seem to flow about as well.
> 
>     Thanks
>      Justin
> 
>     _______________________________________________
>     cisco-nsp mailing list  cisco-nsp at puck.nether.net
>     <mailto:cisco-nsp at puck.nether.net>
>     https://puck.nether.net/mailman/listinfo/cisco-nsp
>     archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> 
> ------------------------------------------------------------------------
> 
> Forget previous vote 