[c-nsp] Re: Rebooting PIX
Mark Taylor
maillist at smashie.ision.co.uk
Thu Feb 24 06:48:20 EST 2005
> Hi,
>
> I've got a Cisco PIX 515 (64Mb RAM) that has started to reboot randomly
> after being up for between 30 mins and 20 hours. It's been fine for a
> couple of years and just started to do this in the last week.
>
> Yesterday I upgraded to software version 6.3(4) in case it was a software
> problem, however it has rebooted again since. The device has been powered
> down/up but that didn't make any difference (although after the power
> cycle it survived the longest interval yet before rebooting).
>
> I'm logging to a syslog server (at warning level), but the logs don't show
> anything to indicate why it would be rebooting and I've made sure that
> there isn't logging to the console. There aren't any interface errors on
> the PIX or on the switch. The config hasn't changed recently and it's not
> down to heat or fan failure as I've checked that. I've also made sure the
> power cables are seated correctly (the site hasn't lost power either).
>
> As far as I can work out, the particular device could potentially suffer
> from any of the following field notices, however I'm not convinced that
> these are the problem because it's uptime was nearly 2 years before this
> problem started about a week ago:
> PIX-515 Defective AC Power Supply
> PIX 515 and 506 Hang
> PIX 515 NMI Exception Crash
>
> I'm just about to leave a PC terminal connected to the console port and
> log it to see if it outputs any console messages before it reboots however
> I'm now beginning to think that this could be a hardware issue.
>
> Has anybody got any other suggestions or ways in which I might be able to
> find the reason that it's rebooting.
>
> Thanks in advance,
> Mark.
Thanks to everybody for their help so far. To confirm the power feed is
clean. I'm still having the issue and managed to capture the output from the
console port this morning when it rebooted again. It seems there was some
type of page fault and then it outputed what you would normally see in "sh
crash" and rebooted.
Not a lot more info to go on unfortunately... I've pasted the console output
from the crash below in case anybody has any thoughts.
Thanks,
Mark.
Thread Name: telnet/ci (Old pc 0x003f608f ebp 0x010c4f48)
Traceback:
0: 004d58cf
1: 004b06b4
2: 00107a84
3: 00107b7e
4: 00103725
5: 00000000
vector 0x0000000e (page fault)
edi 0x000005a0
esi 0x010bdbe0
ebp 0x010c50d8
esp 0x010c50a0
ebx 0x010c5810
edx 0x343f0b44
ecx 0xccccd09c
eax 0xcccccccc
error code 0x00000000
eip 0x004d4201
cs 0x00000008
eflags 0x010c50d8
CR2 0x343f0b4c
Stack dump: base:0x010c35f4 size:8192, active:1492
0x010c55f0: 0x00519f5c
0x010c55ec: 0x00107b6c
0x010c55e8-0x010c55e4: 0x00000000
0x010c55e0-0x010c55c4: 0x12345678
0x010c55c0: 0x00519f5c
0x010c55bc: 0x00103725
0x010c55b8: 0x010c55e4
0x010c55b4-0x010c55a4: 0x12345678
0x010c55a0: 0x00519f5c
0x010c559c: 0x00107b7e
0x010c5598: 0x010c55b8
0x010c5594-0x010c5538: 0x12345678
0x010c5534: 0x12002023
0x010c5530: 0x3030676f
0x010c552c: 0x642d656e
0x010c5528: 0x72756f62
0x010c5524-0x010c532c: 0x00000000
0x010c5328: 0x0000000a
0x010c5324-0x010c5320: 0x00000000
0x010c531c-0x010c52e0: 0x12345678
0x010c52dc: 0x12345600
0x010c52d8-0x010c52bc: 0x12345678
0x010c52b8: 0x12345600
0x010c52b4-0x010c52b0: 0x00000000
0x010c52ac: 0x0051baec
0x010c52a8: 0x12345600
0x010c52a4-0x010c5234: 0x00000000
0x010c5230: 0x00000035
0x010c522c: 0x315f656c
0x010c5228: 0x62616e65
0x010c5224: 0x00332f68
0x010c5220: 0x1011111f
0x010c521c: 0x00507488
0x010c5218: 0x00567508
0x010c5214: 0x00519f5c
0x010c5210: 0x12345600
0x010c520c-0x010c519c: 0x00000000
0x010c5198: 0x00000035
0x010c5194: 0x315f656c
0x010c5190: 0x62616e65
0x010c518c-0x010c5184: 0x00000000
0x010c5180: 0x01345678
0x010c517c: 0x010bae1c
0x010c5178: 0x010bb464
0x010c5174: 0x00000001
0x010c5170: 0x010c5183
0x010c516c: 0x010bae3c
0x010c5168: 0x0010740c
0x010c5164: 0x010c5528
0x010c5160-0x010c515c: 0x12345678
0x010c5158-0x010c5150: 0x00000000
0x010c514c: 0x010c5184
0x010c5148: 0x00000000
0x010c5144: 0x84c00006
0x010c5140: 0x010bae1c
0x010c513c: 0x00107a84
0x010c5138: 0x010c5598
0x010c5134: 0x85b00003
0x010c5130: 0x007293a4
0x010c512c: 0x00107a66
0x010c5128-0x010c5124: 0x010c5190
0x010c5120: 0x010c5184
0x010c511c: 0x00101759
0x010c5118: 0x00000000
0x010c5114: 0x85b00003
0x010c5110: 0x010bd824
0x010c510c: 0x004b06b4
0x010c5108: 0x010c5138
0x010c5104: 0x010c5528
0x010c5100: 0x010c5184
0x010c50fc: 0x004b0f88
0x010c50f8: 0x00000000
0x010c50f4: 0x84c00006
0x010c50f0: 0x00000010
0x010c50ec: 0x002dc05a
0x010c50e8: 0x010c5118
0x010c50e4: 0x00000000
0x010c50e0: 0x010bd818
0x010c50dc: 0x004d58cf
0x010c50d8: 0x010c5108
0x010c50d4: 0x343f0b44
0x010c50d0: 0x000003d0
0x010c50cc: 0x00000000
0x010c50c8: 0x010c50e8
0x010c50c4: 0x010c50e7
0x010c50c0: 0x007293a4
0x010c50bc: 0x004e461d
0x010c50b8: 0x010bae1c
0x010c50b4: 0x010bb318
0x010c50b0: 0x010bd818
0x010c50ac: 0x00010286
0x010c50a8: 0x00000008
0x010c50a4: 0x004d4201
0x010c50a0: 0x00000000 *
0x010c509c: 0xcccccccc
0x010c5098: 0xccccd09c
0x010c5094: 0x343f0b44
0x010c5090: 0x010c5810
0x010c508c: 0x010c50a0
0x010c5088: 0x010c50d8
0x010c5084: 0x010bdbe0
0x010c5080: 0x000005a0
0x010c507c: 0x0000000e
0x010c5078: 0x00104f21
0x010c5074: 0x010c50d8
0x010c5070: 0x00000008
0x010c506c: 0x004d4201
0x010c5068: 0x00000000
0x010c5064: 0xcccccccc
0x010c5060: 0xccccd09c
0x010c505c: 0x343f0b44
0x010c5058: 0x010c5810
0x010c5054: 0x010c50a0
0x010c5050: 0x010c50d8
0x010c504c: 0x010bdbe0
0x010c5048: 0x000005a0
0x010c5044: 0x0000000e
0x010c5040: 0x010c5044
0x010c503c: 0x001016f9
0x010c5038: 0x00000000
0x010c5034: 0x000005a0
0x010c5030: 0x010bdbe0
0x010c502c: 0x010c5810
0x010c5028: 0x010c4f48
0x010c5024: 0x00000000
0x010c5020: 0x00000005
Cisco PIX Firewall Version 6.3(4)
Compiled on Fri 02-Jul-04 00:07 by morlee
bourne-dog00 up 21 hours 35 mins
Hardware: PIX-515, 64 MB RAM, CPU Pentium 200 MHz
Flash i28F640J5 @ 0x300, 16MB
BIOS Flash AT29C257 @ 0xfffd8000, 32KB
0: ethernet0: address is 0050.54ff.fb04, irq 10
1: ethernet1: address is 0050.54ff.fb05, irq 7
2: ethernet2: address is 00d0.b7af.cfdf, irq 9
Licensed Features:
Failover: Enabled
VPN-DES: Enabled
VPN-3DES-AES: Disabled
Maximum Physical Interfaces: 6
Maximum Interfaces: 10
Cut-through Proxy: Enabled
Guards: Enabled
URL-filtering: Enabled
Inside Hosts: Unlimited
Throughput: Unlimited
IKE peers: Unlimited
This PIX has an Unrestricted (UR) license.
Serial Number: 123456789 (0x11111111)
Running Activation Key: 0x11111111 0x11111111 0x11111111 0x11111111
Configuration last modified by enable_15 at 14:31:17.676 UTC Wed Feb 23 2005
------------------ show clock ------------------
09:47:19.871 UTC Thu Feb 24 2005
------------------ show memory ------------------
Free memory: 49411136 bytes
Used memory: 17697728 bytes
------------- ----------------
Total memory: 67108864 bytes
------------------ show conn count ------------------
130 in use, 249 most used
------------------ show xlate count ------------------
74 in use, 203 most used
------------------ show blocks ------------------
SIZE MAX LOW CNT
4 1600 1593 1600
80 400 398 400
256 1012 1002 1012
1550 1317 894 932
------------------ show interface ------------------
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0050.54ff.fb04
IP address 195.7.249.2, subnet mask 255.255.255.224
MTU 1500 bytes, BW 100000 Kbit full duplex
2973218 packets input, 883920159 bytes, 0 no buffer
Received 6382 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2905877 packets output, 1033072237 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/22)
output queue (curr/max blocks): hardware (0/20) software (0/1)
interface ethernet1 "inside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0050.54ff.fb05
IP address 10.153.10.1, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
4509614 packets input, 626785311 bytes, 0 no buffer
Received 23978 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
7073038 packets output, 4213377479 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/20)
output queue (curr/max blocks): hardware (0/22) software (0/1)
interface ethernet2 "dmz1" is up, line protocol is up
Hardware is i82559 ethernet, address is 00d0.b7af.cfdf
IP address 195.7.249.65, subnet mask 255.255.255.224
MTU 1500 bytes, BW 100000 Kbit full duplex
7399372 packets input, 4102271195 bytes, 0 no buffer
Received 1929 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
4873860 packets output, 364402149 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/10)
output queue (curr/max blocks): hardware (0/8) software (0/1)
------------------ show cpu usage ------------------
CPU utilization for 5 seconds = 8%; 1 minute: 2%; 5 minutes: 2%
------------------ show process ------------------
PC SP STATE Runtime SBASE Stack Process
Hrd 001ecf11 00950374 00565bc0 0 0094f3ec 3620/4096 arp_timer
Lsi 001f26c5 009f356c 00565bd8 0 009f25f4 3832/4096 FragDBGC
Lwe 00119aef 009ff714 00569340 0 009fe8ac 3688/4096 dbgtrace
Lwe 003f27e5 00a018a4 0055e510 4630 009ff95c 6880/8192 Logger
Hrd 003f6935 00a0499c 00565bc0 0 00a02a24 7896/8192 tcp_fast
Hrd 003f67d5 00a06a4c 00565bc0 0 00a04ad4 7896/8192 tcp_slow
Lsi 0030d391 00b3d1cc 00565bd8 0 00b3c244 3848/4096 xlate clean
Lsi 0030d29f 00b3e26c 00565bd8 0 00b3d2f4 3884/4096 uxlate clean
Mwe 00304a63 00cd666c 00565bd8 0 00cd46d4 7780/8192
tcp_intercept_timer_process
Lsi 0044a055 00d82f44 00565bd8 0 00d81fbc 3816/4096 route_process
Hrd 002f45ac 00d83fd4 00565bc0 0 00d8306c 2748/4096 PIX Garbage
Collector
Hwe 0021a459 00d8e504 00565bd8 0 00d8a59c 16048/16384
isakmp_time_keeper
Lsi 002f214c 00da8594 00565bd8 0 00da760c 3896/4096 perfmon
Mwe 00211229 00dd29c4 00565bd8 0 00dd0a4c 7816/8192 IPsec timer
handler
Hrd 003a8933 00de7514 00565bc0 0 00de55cc 6920/8192
qos_metric_daemon
Mwe 00266705 00e0204c 00565bd8 10 00dfe0e4 15464/16384 IP Background
Lwe 003056da 00eb4d5c 0057c180 0 00eb3ee4 3704/4096 pix/trace
Lwe 00305912 00eb5e0c 0057c8b0 0 00eb4f94 3704/4096 pix/tconsole
Hrd 0011f247 00ec1ce4 00565bc0 4680 00ebe2fc 12928/16384 ci/console
Hrd 004434da 00ec3b14 00565bc0 10 00ec2bdc 3684/4096 lu_ctl
Crd 002fd61f 00ec4be4 00566050 0 00ec3c8c 3416/4096
update_cpu_usage
Hwe 002e8a81 00f7588c 00544f60 0 00f71a04 15884/16384 uauth_in
Hwe 003f53ed 00f7798c 008c7eb8 0 00f75ab4 7896/8192 uauth_thread
Hwe 0040c7fa 00f78adc 0055eb10 0 00f77b64 3960/4096 udp_timer
Hrd 001e4a0e 00f7a79c 00565bc0 0 00f79824 3800/4096 557mcfix
Crd 001e49c3 00f7b85c 00566050 49988780 00f7a8d4 3580/4096 557poll
Lsi 001e4a7d 00f7c8fc 00565bd8 0 00f7b984 3720/4096 557timer
Cwe 001e6619 00f929d4 0073def0 94450 00f90adc 5736/8192 pix/intf0
Mwe 0040c56a 00f93ae4 00911a28 0 00f92bac 3896/4096 riprx/0
Mrd 003b1ee9 00f94bf4 00565c10 0 00f93c7c 3888/4096 riptx/0
Cwe 001e6619 00f9adfc 007b3480 139980 00f98f04 5736/8192 pix/intf1
Mwe 0040c56a 00f9bf0c 009119e0 0 00f9afd4 3896/4096 riprx/1
Mrd 003b1ee9 00f9d01c 00565c10 0 00f9c0a4 3888/4096 riptx/1
Cwe 001e6619 00fa3224 00828a18 221030 00fa132c 6540/8192 pix/intf2
Mwe 0040c56a 00fa4334 00911998 0 00fa33fc 3896/4096 riprx/2
Mrd 003b1ee9 00fa5444 00565c10 0 00fa44cc 3800/4096 riptx/2
Hrd 00444321 00fa76a4 00565bc0 0 00fa672c 3800/4096 lu_xmit_timer
Hwe 00443005 00fa8744 005622c0 0 00fa77dc 3900/4096 lu_rx
Hrd 001b30f5 00fe3854 00565bc0 0 00fe28ec 2844/4096 fover_thread
Hrd 0011f247 00fe45ac 00565bc0 0 00fe3904 3076/4096 fover_rx
Hwe 001b5ea1 00fe5894 0056dedc 0 00fe491c 3960/4096 fover_tx
Hwe 001b3330 00fe78ac 0056dee8 0 00fe5934 8056/8192 fover_rep
Lwe 001b34ed 00fe88d4 0056def0 0 00fe794c 3976/4096 fover_lu_rep
Hrd 001b64d2 00fec8dc 00565bc0 0 00fe8964 16084/16384 fover_parse
Hwe 003f5681 0105f21c 0089e640 0 0105ebd4 1196/2048 listen/telnet_1
Hwe 003f5681 0105facc 0089e738 0 0105f484 1196/2048 listen/telnet_2
Mwe 0037eb66 01062024 00565bd8 0 010600ac 7960/8192 Crypto CA
H* 003f608f 0009ff2c 00565bc0 420 010c35f4 3908/8192 telnet/ci
------------------ show failover ------------------
Failover Off
Cable status: My side not connected
Reconnect timeout 0:00:00
Poll frequency 15 seconds
------------------ show traffic ------------------
outside:
received (in 77765.720 secs):
2973242 packets 883920159 bytes
38 pkts/sec 11035 bytes/sec
transmitted (in 77765.720 secs):
2905877 packets 1033072237 bytes
37 pkts/sec 13008 bytes/sec
inside:
received (in 77765.940 secs):
4509657 packets 626785311 bytes
2 pkts/sec 8004 bytes/sec
transmitted (in 77765.940 secs):
7073038 packets 4213377479 bytes
35 pkts/sec 54014 bytes/sec
dmz1:
received (in 77766.150 secs):
7399374 packets 4102271195 bytes
39 pkts/sec 52033 bytes/sec
transmitted (in 77766.150 secs):
4873860 packets 364402149 bytes
7 pkts/sec 4023 bytes/sec
------------------ show perfmon ------------------
PERFMON STATS: Current Average
Xlates 0/s 0/s
Connections 1/s 1/s
TCP Conns 1/s 0/s
UDP Conns 0/s 0/s
URL Access 0/s 0/s
URL Server Req 0/s 0/s
TCP Fixup 486/s 22/s
TCPIntercept 0/s 0/s
HTTP Fixup 20/s 6/s
FTP Fixup 2/s 2/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s
------------------ show running-config ------------------
: Saved
:
PIX Version 6.3(4)
*CONFIG GOES HERE BUT OUTPUT TERMINTATED EARLY*
-----------------------------------------------
Nested traceback attempted via watchdog.
Traceback output aborted.
-----------------------------------------------
Rebooting....
Cisco Secure PIX Firewall BIOS (4.0) #0: Thu Mar 2 22:59:20 PST 2000
Platform PIX-515
Flash=i28F640J5 @ 0x300
Use BREAK or ESC to interrupt flash boot.
Use SPACE to begin flash boot immediately.
Reading 1962496 bytes of image from flash.
imgsum_config: sumval(0x12b1) md5(0x96a6bc99 0xfdcbe539 0x38a4f658
0x28d53468)
imgsum_verify: chksum(0x0 ) md5(0xad0f6296 0xb736cf05 0x42a852a7
0x24e120b1)
Checksum verification on compression loader failed.
Rebooting....
Cisco Secure PIX Firewall BIOS (4.0) #0: Thu Mar 2 22:59:20 PST 2000
Platform PIX-515
Flash=i28F640J5 @ 0x300
Use BREAK or ESC to interrupt flash boot.
Use SPACE to begin flash boot immediately.
Reading 1962496 bytes of image from flash.
###############################################################################################################
64MB RAM
mcwa i82559 Ethernet at irq 10 MAC: 0050.54ff.fb04
mcwa i82559 Ethernet at irq 7 MAC: 0050.54ff.fb05
mcwa i82559 Ethernet at irq 9 MAC: 00d0.b7af.cfdf
Flash=i28F640J5 @ 0x300
BIOS Flash=AT29C257 @ 0xfffd8000
More information about the cisco-nsp
mailing list