[rbak-nsp] Queue and Nat errors

Fri Apr 4 16:39:38 EDT 2014

Hi
on se100 SEOS-11.1.2.6-Release
we suffer from a huge huge number of logs like

Apr 4 08:52:19: %PPAIP-3-NAT_ERROR: 
fa934af2/0000052909/000900000:02/IPPA/EU00:NAT: outhash delete failed. 
entry (50ada900) ip (a01056f:674b) nat (5be1bd06:12b2) lock (ce6c0f) - 
000723cc
000e3d0c 00071f34 0013c422 0013c8bc 0013d036 000df704 000df6dc 00000000

Apr 4 15:49:59: %PPAINFRA-3-QUEUE_ERROR: 
cbb8d208/0000077968/591000000:02/IPPA/EU00:Attempt to enqueue singly 
linked element 0x6e1ed2c0 next 0x000066c3 - 000723cc 000e3d0c 00071f34 
000f35
0c 000f3572 00141e94 001383a0 00140f1a 0013f408 001357f8 0013f984 
000f18aa 0013f920 000df70

Apr 4 19:52:07: %PPAINFRA-3-QUEUE_ERROR: 
564027b1/0000092496/505100000:02/IPPA/EU00:Attempt to enqueue singly 
linked element 0x6e1da460 next 0xc5b50000 - 000723cc 000e3d0c 00071f34 
000f35
0c 000f3572 00141e94 001383a0 00140f1a 0013f408 001357f8 0013f984 
000f18aa 0013f920 000df70
Apr 4 19:52:08: %PPAINFRA-3-QUEUE_ERROR: 
677eb061/0000092497/287000000:02/IPPA/EU00: [27 dropped]: Attempt to 
enqueue singly linked element 0x5e23c100 next 0x80020000 - 000723cc 
000e3d0c
00071f34 000f350c 000f3572 00141e94 001383a0 00140f1a 00078850 00078956 
000df704 000df6dc 
0000`<E8>^A<FE>S><FF>X!W<CF><F8>^?<FF><FF><FF><FF><FF><FF><FF><FF><FF><FF><FF>

What can be reason of these logs ?
We also had situation that multiple pppoe sessions were broken. But we 
do not know if it was the reason of these logs or the result:/

Two days ago we had crash

Apr 3 18:04:33: %CSM-6-CARD: PPA crash detected on card carrier 2
Apr 3 18:04:33: %CSM-6-PORT: ethernet 2/1 link state DOWN service state 
DOWN, overall admin is UP
Apr 3 18:04:33: %CSM-6-PORT: ethernet 2/1 link state down, trigger 
source: Configuration changed or hardware removed
Apr 3 18:04:33: %CSM-6-PORT: ethernet 2/2 link state DOWN service state 
DOWN, overall admin is UP
Apr 3 18:04:33: %CSM-6-PORT: ethernet 2/2 link state down, trigger 
source: Configuration changed or hardware removed
Apr 3 18:04:34: %PM-1-ALERT: Process IPPA IPC SLOT 2, pid 0x80020001, 
is not responding.
Apr 3 18:04:37: %SYSLOG-2-CRIT: /netbsd: xcrp_output() - returning 
ENETDOWN, fwd_cap = 2, lc_hash_fails = 0
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/3 link state DOWN service state 
DOWN, overall admin is UP
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/3 link state down, trigger 
source: Port fault detected
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/4 link state DOWN service state 
DOWN, overall admin is UP
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/4 link state down, trigger 
source: Port fault detected
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/15 link state DOWN service 
state DOWN, overall admin is UP
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/15 link state down, trigger 
source: Port fault detected
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/16 link state DOWN service 
state DOWN, overall admin is UP
Apr 3 18:04:37: %CSM-6-PORT: ethernet 2/16 link state down, trigger 
source: Port fault detected
Apr 3 18:04:49: %PPAINFRA-3-LOG_ERR: crashSrvr(): Slot/ppa 2/0 Crash 
server data collection started

Apr 3 18:09:38: %PPAINFRA-3-LOG_ERR: crashSrvr(): Slot/ppa 2/0 Crash 
server data collection done
Apr 3 18:09:42: %SYSMON-5-GEN_FTP: Core file 
/ata0/p02/ppa/crashSlot02Ippa.gz FTPed to /md/ successfully
Apr 3 18:10:07: %SYSMON-5-GEN_FTP: Core file 
/ata0/p02/ppa/crashSlot02IppaCore.gz FTPed to /md/ successfully
Apr 3 18:10:08: %CSM-6-CARD: card carrier REMOVED from slot 2
Apr 3 18:10:14: %PM-1-ALERT: Process EPPA IPC SLOT 2, pid 0x81020001, 
is not responding.
Apr 3 18:10:18: %CSM-6-CARD: card carrier INSERTED in slot 2
Apr 3 18:10:24: %CSM-6-CARD: MIC number 1 ge-2-port REMOVED from slot 2
Apr 3 18:10:24: %SNMP-6-INFO: The last entity configuration change 
notification was sent out 2.00 (less than 5.00) seconds before. 
Suppressing this one.
Apr 3 18:10:26: %CSM-6-CARD: MIC number 2 ge-2-port REMOVED from slot 2
Apr 3 18:10:26: %SNMP-6-INFO: The last entity configuration change 
notification was sent out 4.00 (less than 5.00) seconds before. 
Suppressing this one.
Apr 3 18:10:26: %CSM-6-CARD: MIC number 1 ge-2-port INSERTED in slot 2
Apr 3 18:10:26: %SNMP-6-INFO: The last entity configuration change 
notification was sent out 4.00 (less than 5.00) seconds before. 
Suppressing this one.
Apr 3 18:10:27: %CSM-6-CARD: MIC number 2 ge-2-port INSERTED in slot 2
Apr 3 18:10:31: %QOS-6-INFO: qos info: PPA just reborn 0
Apr 3 18:10:31: %PPAINFRA-6-ISTART_INFO: 
2b415603/0000000001/854400000:02/IPPA/EU00:Ready to receive packets
Apr 3 18:10:31: %QOS-6-INFO: qos info: iPPA reg on slot 2
Apr 3 18:10:32: %CSM-6-CARD: card carrier INSERTED in slot 2 READY
Apr 3 18:10:32: %PPAINFRA-6-ISTART_INFO: 
36d7b202/0000000002/379900000:02/EPPA/EU00:Ready to receive packets
Apr 3 18:10:32: %CSM-6-PORT: ethernet 2/4 link state UP service state 
UP, overall admin is UP
Apr 3 18:10:32: %CSM-6-PORT: ethernet 2/15 link state UP service state 
UP, overall admin is UP
Apr 3 18:10:32: %CSM-6-PORT: ethernet 2/16 link state UP service state 
UP, overall admin is UP
Apr 3 18:10:32: %QOS-6-INFO: qos info: PPA just reborn 0
Apr 3 18:10:32: %QOS-6-INFO: qos info: ePPA reg on slot 2
Apr 3 18:10:33: %CSM-6-CARD: Card in slot 2 entering In Service state.
Apr 3 18:10:34: %PPAINFRA-6-IFACE_INFO: 
6c5fbbf8/0000000004/807100000:02/IPPA/EU00:ROUTING_READY sent to CSM
Apr 3 18:10:34: %CSM-6-PORT: ethernet 2/1 link state UP service state 
UP, overall admin is UP
Apr 3 18:10:35: %CSM-6-PORT: ethernet 2/2 link state UP service state 
UP, overall admin is UP
Apr 3 18:10:37: %CSM-6-PORT: ethernet 2/3 link state UP service state 
UP, overall admin is UP

What can be the reason ? Please any advice - any help.

[local]redback#show memory
Memory: Total 802720k, Used 617676k, Free 142884k, Reserved 24k
[local]redback#

[local]redback#show crashfiles
      1292101 Apr  3 18:09 /md/20140403_180942_crashSlot02Ippa.gz
     19282148 Apr  3 18:10 /md/20140403_181007_crashSlot02IppaCore.gz

-- 
Pozdrawiam
"Kuba" Dawid Chrzan
pszczyna.net.pl
+48503038479