[c-nsp] Router down - What is "Net Background"?
Justin Shore
justin at justinshore.com
Fri Apr 4 11:47:37 EDT 2008
Thanks for all the replies. One of our OC3s had the FREF light on,
sometimes flashing too. I started pulling IMA T1s to see if it was
customer traffic. None of them fixed the problem. When I pulled that
particular OC3 the problem cleared up.
I did see flaps in the log for that particular OC3. They were happening
approximately every 60s. Looking at the log a little closer I see that
ATM3/0 would come up and then drop within 3-4s, wait 60s, rinse and repeat.
I have on that OC3 about 300 PVCs. None of the PVCs have a ubr over
1664. We're using RBE instead of PPPoX. I have ingress & egress ACLs
on each PVC as well as uRPF. The vast majority of the PVCs are
unnumbered to a loopback, though a few have /30s. Other than that the
interface config is basic. There isn't any QoS or shaping going on,
other than setting the ubr.
When I started disconnecting circuits I was speculating that customer
traffic could have been to blame. In the log I noticed repeating ACL
drops from RFC1918 IPs to our name servers. There weren't that many
hits on the ACL but there were some occasional ACL rate-limit overruns.
I've seen far worse though. I can't remember if ACLs happen in HW on
the 3660 or not; even if they do ACL logging is a process level function
isn't it?
Every time that OC3 bounces that's about 300 sub-ints bouncing with it.
I imagine repeated bounces would be a painful thing. I've pulled the
carrier's OC3 card to get the 3660 back online. I'm planning on
shutting down all 300 sub-ints on that OC3 and bring the OC3 back up.
If the 3660 is still ok I'll enable about 50 PVCs at a time until the
box either flips out or I have all the PVCs back online. I have a spare
carrier card but I don't think I have a spare OC3 NM so hopefully that's
not the problem.
Thanks for all the input.
Justin
Christian wrote:
> the net background proess. sends interface keepalive packets and
> processes interface state changes
>
> On Fri, Apr 4, 2008 at 10:42 AM, Justin Shore <justin at justinshore.com
> <mailto:justin at justinshore.com>> wrote:
>
> I just had a 3660 terminating about 1500 PVCs for DSL die this AM. The
> 3660 is running 12.3(22) w/ 256MB of RAM. I'm at the remote POP now,
> consoled in and looking at the box. The console hangs for 45-60s before
> I get a chance to run a quick command or two. Then it goes back to the
> hung state about 5 seconds later. The box keeps dropping my 2 OC3s and
> bouncing my IS-IS processes. Syslog didn't appear to capture anything
> interesting leading up to this crash. I had the box power cycled while
> I was driving down here.
>
> Overall the CPU is low and neither CPU nor memory were being taxed over
> the past number of months. The Net Background process is using a fair
> bit of the processor. Actually, that's an understatement. At times
> it's only using 9% or so. At other times it's up to 97% as was the case
> here:
>
> 3660-2.brd#sh proc cpu sort 5m
> CPU utilization for five seconds: 1%/0%; one minute: 17%; five
> minutes: 19%
> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> 28 4715940 611 7731049 97.09% 13.74% 9.23% 0 Net
> Background
> 31 238608 865 275847 2.65% 8.58% 8.48% 0
> Per-Second Jobs
> 5 8336 684 12187 0.00% 0.56% 0.68% 0 Check
> heaps
> 60 7764 7776 998 0.01% 0.27% 0.43% 0 IP Input
>
> 18 3856 8741 441 0.00% 0.42% 0.33% 0 ARP
> Input
> 167 2556 3046 839 0.02% 0.12% 0.13% 0 ISIS Upd
>
> 140 2732 15 182133 0.00% 0.00% 0.12% 0 Key Proc
>
> 179 1680 1676 1002 0.00% 0.13% 0.11% 0 OSPF
> Router
> 47 1912 152 12578 0.00% 0.12% 0.11% 0
> Per-minute Jobs
> 82 1792 597 3001 0.01% 0.15% 0.11% 0 IP
> Background
> 3 1828 238 7680 0.00% 0.11% 0.09% 0 OSPF
> Hello
> 100 644 1217 529 0.00% 0.13% 0.08% 0 DHCPD
> Receive
> 51 1700 376 4521 0.00% 0.04% 0.05% 0 ATM
> Periodic
> 83 700 417 1678 0.00% 0.01% 0.02% 0 IP RIB
> Update
> 108 500 871 574 0.00% 0.06% 0.02% 0 CEF
> process
> 124 172 471 365 0.00% 0.04% 0.02% 0 Exec
>
> 166 620 1908 324 0.00% 0.03% 0.02% 0 ISIS Adj
>
> 106 420 284 1478 0.00% 0.03% 0.00% 0 Adj
> Manager
> 6 436 1406 310 0.00% 0.02% 0.00% 0 Pool
> Manager
> 91 188 556 338 0.00% 0.01% 0.00% 0 ILMI
> Timer Proce
> 163 92 1795 51 0.00% 0.00% 0.00% 0 CLNS
> Input
>
>
> Per-Second Jobs is also using quite a bit of CPU. The log has CPUHOG
> entries in it pointing at the Net Background process too:
>
> 000428: .Apr 4 08:36:31 CDT: %SYS-3-CPUHOG: Task ran for 50332 msec
> (10/0), process = Net Background, PC = 60439FFC.
> -Traceback= 6043A004
>
> What does the Net Background process do? I haven't been able to find
> any answers on Cisco.com.
>
>
> We're in the middle of SmartNet renewals and negotiations and
> unfortunately this box's contract has expired. I can have our AM bless
> a TAC case for us if need be, assuming I can locate him; I think he's in
> training this week back East.
>
> Any ideas? I'm going to go back and scrutinize the log for anything
> useful. As of right now the box is intermittently pingable,
> corresponding directly to the brief instances that I can use the
> console. Customer packets seem to flow about as well.
>
> Thanks
> Justin
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> <mailto:cisco-nsp at puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
More information about the cisco-nsp
mailing list