[nsp] Router dumping core

Tue Sep 23 18:45:43 EDT 2003

Looks to me like you're being got by one of the recent IOS security
vulnerabilities.  Exploit code went into the wild a few weeks ago and
various strange packets have been bouncing around.

Upgrade your code!

Simon

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net]On Behalf Of james
Sent: Tuesday, September 23, 2003 4:48 PM
To: cisco-nsp at puck.nether.net
Cc: gordon@[198.59.109.2]; msanchez@[198.59.109.2];
dlacey at cybermesa.com; jamesh at mail.cybermesa.com
Subject: [nsp] Router dumping core

I have a 7206, lightly loaded (less than 10% cpu), which is locking up and
dumping the core.
Twice in the last 3 weeks. Logs are unremarkable (till today) and the router
has been in service for several years with no problems. I syslog to a remote
syslog-ng server but the router only gets out a line or so of errors and
stops forwarding packets out of all interfaces. Even the Ethernet.
I did not get the reboot the router myself after the 2 crashes &
the staff did not save the routers logging buffer.

I have seen this in the log, prior to the lockup (both times) , but I am not
running console debugging, "term mon" & "no
logging console" is
in config & tehre is no debugging sessions running:

Sep 22 21:52:34 sfr-ser 15284: Sep 22 21:13:21.764 MDT:
%SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure
console debugging output.

Today it is starting to look bad :

[root at tarpit routerlogs]# tail -n 400 sf-router.log | grep -A 4
%SYS-2-MALLOCFAIL:
Sep 23 06:14:31 sfr-ser 76: Sep 23 06:14:30.135 MDT: %SYS-2-MALLOCFAIL:
Memory allocation of 65556 bytes failed from
0x6064465C, pool I/O-2, alignment 32
Sep 23 06:14:31 sfr-ser 77: -Process= "Pool Manager", ipl= 0, pid= 5
Sep 23 06:14:31 sfr-ser 78: -Traceback= 60648B80 6064A470 60644664 60624A44
60656CA8 6063D8BC 6063D8A8
Sep 23 06:15:22 sfr-ser 79: Sep 23 06:15:21.292 MDT: %SYS-2-MALLOCFAIL:
Memory allocation of 65556 bytes failed from
0x6064465C, pool I/O-2, alignment 32
Sep 23 06:15:22 sfr-ser 80: -Process= "Per-minute Jobs", ipl= 0, pid= 98
Sep 23 06:15:22 sfr-ser 81: -Traceback= 60648B80 6064A470 60644664 60624A44
60657780 60624E3C 6065EE34 606135F0 6063D8BC
6063D8A8
Sep 23 06:15:48 sfr-ser 82: Sep 23 06:15:48.624 MDT: %SYS-3-CPUHOG: Task ran
for 13016 msec (128/40), process = Pool
Manager, PC = 60656BB0.
Sep 23 06:15:48 sfr-ser 83: -Traceback= 60656BB8 6063D8BC 6063D8A8
--
Sep 23 06:21:04 sfr-ser 94: Sep 23 06:21:03.842 MDT: %SYS-2-MALLOCFAIL:
Memory allocation of 65556 bytes failed from
0x6064465C, pool I/O-2, alignment 32
Sep 23 06:21:04 sfr-ser 95: -Process= "Pool Manager", ipl= 0, pid= 5
Sep 23 06:21:04 sfr-ser 96: -Traceback= 60648B80 6064A470 60644664 60624A44
60656CA8 6063D8BC 6063D8A8
Sep 23 06:22:17 sfr-ser 97: Sep 23 06:22:17.398 MDT: %SYS-3-CPUHOG: Task ran
for 13348 msec (140/48), process = Pool
Manager, PC = 60656BB0.
Sep 23 06:22:17 sfr-ser 98: -Traceback= 60656BB8 6063D8BC 6063D8A8
[root at tarpit routerlogs]#

Questions:

1) Any clues to what is failing/being bad ?
2) I have run strings on the core dumps, is there a IOS debugger for this ?
3) I have a local term server, I would like to log out the AUX port to the
term server
but am not finding the commands to do this. As ethernet fails, also, I am
looking for a low level
way to continue sys-logging.

sf-atm>sho ver
Cisco Internetwork Operating System Software
IOS (tm) 7200 Software (C7200-P-M), Version 12.2(6), RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Wed 07-Nov-01 22:40 by pwade
Image text-base: 0x600089C0, data-base: 0x6112C000

ROM: System Bootstrap, Version 12.0(19990210:195103) [12.0XE 105],
DEVELOPMENT SOFTWARE
BOOTLDR: 7200 Software (C7200-BOOT-M), Version 12.0(10)S, EARLY DEPLOYMENT
RELEASE SOFTWAR
E (fc1)

sf-atm uptime is 19 hours, 37 minutes
System returned to ROM by power-on
System restarted at 22:00:09 MDT Mon Sep 22 2003
System image file is "slot0:c7200-p-mz.122-6.bin"

cisco 7206VXR (NPE300) processor (revision D) with 229376K/65536K bytes of
memory.
Processor board ID 20391418
R7000 CPU at 262Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 Cache
6 slot VXR midplane, Version 2.0

Last reset from power-on
X.25 software, Version 3.0.0.
Bridging software.
Primary Rate ISDN software, Version 1.1.
1 FastEthernet/IEEE 802.3 interface(s)
9 Serial network interface(s)
1 ATM network interface(s)
8 Channelized T1/PRI port(s)
125K bytes of non-volatile configuration memory.

20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
4096K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x2102

sf-atm>sho proc cpu | exclude 0.00
CPU utilization for five seconds: 3%/2%; one minute: 2%; five minutes: 2%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  41       36796    414769         88  0.24%  0.10%  0.07%   0 IP Input
  79          36        93        387  0.49%  0.04%  0.01%   2 Virtual Exec
sf-atm>

sf-atm>sho mem
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)
Largest(b)
Processor   61D8DFA0   203890784    12518508   191372276   183622696
191291716
      I/O   20000000    33554432      348120    33206312       12728
33206172
    I/O-2    E000000    33554432     3752088    29802344       19952
29802300

James Edwards
Routing and Security
jamesh at cybermesa.com
At the Santa Fe Office: Internet at Cyber Mesa
Store hours: 9-6 Monday through Friday
Phone support 365 days till 10 pm via the Santa Fe office:
505-988-9200 or Toll Free: 888-988-2700
SIP:1(747)669-1965

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/