[nsp] Router dumping core

Tue Sep 23 21:46:22 EDT 2003

You are running out of memory on the box.  Check your config, are you 
getting swamped with ICMPS?  Are you blocking ICMPS using route-maps as 
Cisco suggets because of Nachi/Blaster ?

-Matt

On Tuesday, September 23, 2003, at 07:47 PM, james wrote:

> I have a 7206, lightly loaded (less than 10% cpu), which is locking up 
> and dumping the core.
> Twice in the last 3 weeks. Logs are unremarkable (till today) and the 
> router
> has been in service for several years with no problems. I syslog to a 
> remote
> syslog-ng server but the router only gets out a line or so of errors 
> and
> stops forwarding packets out of all interfaces. Even the Ethernet.
> I did not get the reboot the router myself after the 2 crashes &
> the staff did not save the routers logging buffer.
>
> I have seen this in the log, prior to the lockup (both times) , but I 
> am not running console debugging, "term mon" & "no
> logging console" is
> in config & tehre is no debugging sessions running:
>
> Sep 22 21:52:34 sfr-ser 15284: Sep 22 21:13:21.764 MDT: 
> %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure
> console debugging output.
>
> Today it is starting to look bad :
>
> [root at tarpit routerlogs]# tail -n 400 sf-router.log | grep -A 4  
> %SYS-2-MALLOCFAIL:
> Sep 23 06:14:31 sfr-ser 76: Sep 23 06:14:30.135 MDT: 
> %SYS-2-MALLOCFAIL: Memory allocation of 65556 bytes failed from
> 0x6064465C, pool I/O-2, alignment 32
> Sep 23 06:14:31 sfr-ser 77: -Process= "Pool Manager", ipl= 0, pid= 5
> Sep 23 06:14:31 sfr-ser 78: -Traceback= 60648B80 6064A470 60644664 
> 60624A44 60656CA8 6063D8BC 6063D8A8
> Sep 23 06:15:22 sfr-ser 79: Sep 23 06:15:21.292 MDT: 
> %SYS-2-MALLOCFAIL: Memory allocation of 65556 bytes failed from
> 0x6064465C, pool I/O-2, alignment 32
> Sep 23 06:15:22 sfr-ser 80: -Process= "Per-minute Jobs", ipl= 0, pid= 
> 98
> Sep 23 06:15:22 sfr-ser 81: -Traceback= 60648B80 6064A470 60644664 
> 60624A44 60657780 60624E3C 6065EE34 606135F0 6063D8BC
> 6063D8A8
> Sep 23 06:15:48 sfr-ser 82: Sep 23 06:15:48.624 MDT: %SYS-3-CPUHOG: 
> Task ran for 13016 msec (128/40), process = Pool
> Manager, PC = 60656BB0.
> Sep 23 06:15:48 sfr-ser 83: -Traceback= 60656BB8 6063D8BC 6063D8A8
> --
> Sep 23 06:21:04 sfr-ser 94: Sep 23 06:21:03.842 MDT: 
> %SYS-2-MALLOCFAIL: Memory allocation of 65556 bytes failed from
> 0x6064465C, pool I/O-2, alignment 32
> Sep 23 06:21:04 sfr-ser 95: -Process= "Pool Manager", ipl= 0, pid= 5
> Sep 23 06:21:04 sfr-ser 96: -Traceback= 60648B80 6064A470 60644664 
> 60624A44 60656CA8 6063D8BC 6063D8A8
> Sep 23 06:22:17 sfr-ser 97: Sep 23 06:22:17.398 MDT: %SYS-3-CPUHOG: 
> Task ran for 13348 msec (140/48), process = Pool
> Manager, PC = 60656BB0.
> Sep 23 06:22:17 sfr-ser 98: -Traceback= 60656BB8 6063D8BC 6063D8A8
> [root at tarpit routerlogs]#
>
> Questions:
>
> 1) Any clues to what is failing/being bad ?
> 2) I have run strings on the core dumps, is there a IOS debugger for 
> this ?
> 3) I have a local term server, I would like to log out the AUX port to 
> the term server
> but am not finding the commands to do this. As ethernet fails, also, I 
> am looking for a low level
> way to continue sys-logging.
>
> sf-atm>sho ver
> Cisco Internetwork Operating System Software
> IOS (tm) 7200 Software (C7200-P-M), Version 12.2(6), RELEASE SOFTWARE 
> (fc2)
> Copyright (c) 1986-2001 by cisco Systems, Inc.
> Compiled Wed 07-Nov-01 22:40 by pwade
> Image text-base: 0x600089C0, data-base: 0x6112C000
>
> ROM: System Bootstrap, Version 12.0(19990210:195103) [12.0XE 105], 
> DEVELOPMENT SOFTWARE
> BOOTLDR: 7200 Software (C7200-BOOT-M), Version 12.0(10)S, EARLY 
> DEPLOYMENT RELEASE SOFTWAR
> E (fc1)
>
> sf-atm uptime is 19 hours, 37 minutes
> System returned to ROM by power-on
> System restarted at 22:00:09 MDT Mon Sep 22 2003
> System image file is "slot0:c7200-p-mz.122-6.bin"
>
> cisco 7206VXR (NPE300) processor (revision D) with 229376K/65536K 
> bytes of memory.
> Processor board ID 20391418
> R7000 CPU at 262Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 
> Cache
> 6 slot VXR midplane, Version 2.0
>
> Last reset from power-on
> X.25 software, Version 3.0.0.
> Bridging software.
> Primary Rate ISDN software, Version 1.1.
> 1 FastEthernet/IEEE 802.3 interface(s)
> 9 Serial network interface(s)
> 1 ATM network interface(s)
> 8 Channelized T1/PRI port(s)
> 125K bytes of non-volatile configuration memory.
>
> 20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
> 4096K bytes of Flash internal SIMM (Sector size 256K).
> Configuration register is 0x2102
>
> sf-atm>sho proc cpu | exclude 0.00
> CPU utilization for five seconds: 3%/2%; one minute: 2%; five minutes: 
> 2%
>  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>   41       36796    414769         88  0.24%  0.10%  0.07%   0 IP Input
>   79          36        93        387  0.49%  0.04%  0.01%   2 Virtual 
> Exec
> sf-atm>
>
> sf-atm>sho mem
>                 Head    Total(b)     Used(b)     Free(b)   Lowest(b)  
> Largest(b)
> Processor   61D8DFA0   203890784    12518508   191372276   183622696   
> 191291716
>       I/O   20000000    33554432      348120    33206312       12728   
>  33206172
>     I/O-2    E000000    33554432     3752088    29802344       19952   
>  29802300
>
> James Edwards
> Routing and Security
> jamesh at cybermesa.com
> At the Santa Fe Office: Internet at Cyber Mesa
> Store hours: 9-6 Monday through Friday
> Phone support 365 days till 10 pm via the Santa Fe office:
> 505-988-9200 or Toll Free: 888-988-2700
> SIP:1(747)669-1965
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>