[cisco-nas] MALLOCFAIL

[Adam Greene]

Hi --

I lost most IP connectivity to our AS5396 running IOS (tm) 5300 Software
(C5300-IS-M), Version 12.0(26) this morning. This is what the logs showed
before / during the outage:

2003-09-20 00:33:55 Local7.Critical 208.29.192.14 18092: -Process= "TTY
Background", ipl= 0, pid= 50
2003-09-20 00:33:55 Local7.Critical 208.29.192.14 18093: -Traceback=
601D066C 601D1A60 601EC698 601E41EC 6015424C 60154968 60166DE4 60167200
60167A28 60167BD8 6017E7B0 601CBA2C 601CBA18
2003-09-20 00:33:55 Local7.Critical 208.29.192.14 18094: *Jan  5 20:56:18:
%SYS-2-CFORKMEM: Process creation of Exec failed (no memory).
2003-09-20 00:33:55 Local7.Critical 208.29.192.14 18095: -Process= "TTY
Background", ipl= 0, pid= 50
2003-09-20 00:33:55 Local7.Critical 208.29.192.14 18096: -Traceback=
601EC700 601E41EC 6015424C 60154968 60166DE4 60167200 60167A28 60167BD8
6017E7B0 601CBA2C 601CBA18
2003-09-20 00:35:01 Local7.Critical 208.29.192.14 18097: *Jan  5 20:57:25:
%SYS-2-MALLOCFAIL: Memory allocation of 95 bytes failed from 0x6045D264,
pool Processor, alignment 0
2003-09-20 00:35:01 Local7.Critical 208.29.192.14 18098: -Process= "IP
SNMP", ipl= 0, pid= 60
2003-09-20 00:35:01 Local7.Critical 208.29.192.14 18099: -Traceback=
601D066C 601D2118 6045D26C 60453F34 6045E4AC 6046DFA0 60271EB8 601CBA2C
601CBA18
2003-09-20 00:35:17 Local7.Notice 208.29.192.14 18100: *Jan  5 20:57:40:
%LINK-5-CHANGED: Interface Async26, changed state to reset
2003-09-20 00:35:20 Local7.Critical 208.29.192.14 18101: *Jan  5 20:57:43:
%SYS-2-CFORKMEM: Process creation of Async tty Reset failed (no memory).
2003-09-20 00:35:20 Local7.Critical 208.29.192.14 18102: -Process= "Serial
Background", ipl= 0, pid= 6
2003-09-20 00:35:20 Local7.Critical 208.29.192.14 18103: -Traceback=
601EC700 601E41EC 603110E8 60311340 6003B964 601CBA2C 601CBA18
2003-09-20 00:35:22 Local7.Error 208.29.192.14 18104: *Jan  5 20:57:46:
%LINK-3-UPDOWN: Interface Async26, changed state to down
2003-09-20 00:35:40 Local7.Critical 208.29.192.14 18105: *Jan  5 20:58:04:
%SYS-2-MALLOCFAIL: Memory allocation of 113 bytes failed from 0x601D2884,
pool Processor, alignment 0
2003-09-20 00:35:40 Local7.Critical 208.29.192.14 18106: -Process= "IP
Input", ipl= 0, pid= 23
2003-09-20 00:35:40 Local7.Critical 208.29.192.14 18107: -Traceback=
601D066C 601D2118 601D288C 602B74E8 602F38F4 60315304 60345F80 6034A1D4
601A3E74 6024E8B4 60242A98 6024353C 60242280 6024245C 602425D0 601CBA2C
2003-09-20 00:36:07 Local7.Critical 208.29.192.14 18108: *Jan  5 20:58:27:
%SYS-2-CFORKMEM: Process creation of Exec failed (no memory).
2003-09-20 00:36:07 Local7.Critical 208.29.192.14 18109: -Process= "TTY
Background", ipl= 0, pid= 50
2003-09-20 00:36:07 Local7.Critical 208.29.192.14 18110: -Traceback=
601EC700 601E41EC 6015424C 60154968 60166DE4 60167200 60167A28 60167BD8
6017E7B0 601CBA2C 601CBA18
2003-09-20 00:36:12 Local7.Critical 208.29.192.14 18111: *Jan  5 20:58:34:
%SYS-2-MALLOCFAIL: Memory allocation of 113 bytes failed from 0x601D2884,
pool Processor, alignment 0
2003-09-20 00:36:12 Local7.Critical 208.29.192.14 18112: -Process= "IP
Input", ipl= 0, pid= 23
2003-09-20 00:36:12 Local7.Critical 208.29.192.14 18113: -Traceback=
601D066C 601D2118 601D288C 602B74E8 602F38F4 60315304 60345F80 6034A1D4
601A3E74 6024E8B4 60242A98 6024353C 60242280 6024245C 602425D0 601CBA2C
2003-09-20 00:37:29 Local7.Critical 208.29.192.14 18114: *Jan  5 20:59:52:
%SYS-2-MALLOCFAIL: Memory allocation of 40 bytes failed from 0x6021B748,
pool Processor, alignment 0
2003-09-20 00:37:29 Local7.Critical 208.29.192.14 18115: -Process= "IP
Input", ipl= 0, pid= 23
2003-09-20 00:37:29 Local7.Critical 208.29.192.14 18116: -Traceback=
601D066C 601D2118 6021B750 60258D28 60259744 602597CC 602595A4 60259640
6025F3F8 60033D2C 601A3E74 6024E8B4 60242A98 6024353C 60242280 6024245C
2003-09-20 00:37:37 Local7.Notice 208.29.192.14 18117: *Jan  5 21:00:00:
%LINK-5-CHANGED: Interface Async96, changed state to reset
2003-09-20 00:37:40 Local7.Critical 208.29.192.14 18118: *Jan  5 21:00:03:
%SYS-2-CFORKMEM: Process creation of Async tty Reset failed (no memory).
2003-09-20 00:37:40 Local7.Critical 208.29.192.14 18119: -Process= "Serial
Background", ipl= 0, pid= 6
2003-09-20 00:37:40 Local7.Critical 208.29.192.14 18120: -Traceback=
601EC700 601E41EC 603110E8 60311340 6003B964 601CBA2C 601CBA18
2003-09-20 00:37:42 Local7.Error 208.29.192.14 18121: *Jan  5 21:00:05:
%LINK-3-UPDOWN: Interface Async96, changed state to down
2003-09-20 00:37:59 Local7.Critical 208.29.192.14 18122: *Jan  5 21:00:22:
%SYS-2-MALLOCFAIL: Memory allocation of 40 bytes failed from 0x6021B748,
pool Processor, alignment 0
2003-09-20 00:37:59 Local7.Critical 208.29.192.14 18123: -Process= "IP
Input", ipl= 0, pid= 23
2003-09-20 00:37:59 Local7.Critical 208.29.192.14 18124: -Traceback=
601D066C 601D2118 6021B750 60258D28 60259744 602597CC 602595A4 60259640
6025F3F8 60033D2C 601A3E74 6024E8B4 60242A98 6024353C 60242280 6024245C
2003-09-20 00:38:29 Local7.Critical 208.29.192.14 18125: *Jan  5 21:00:52:
%SYS-2-MALLOCFAIL: Memory allocation of 40 bytes failed from 0x6021B748,
pool Processor, alignment 0
2003-09-20 00:38:29 Local7.Critical 208.29.192.14 18126: -Process= "IP
Input", ipl= 0, pid= 23

Any ideas as to what I should do to start troubleshooting this beast?

TIA,
Adam

---
[This e-mail was scanned for viruses by Webjogger's AntiVirus Protection System]