[j-nsp] M20 stopped forwarding, restart of SSB-E16 necessary

[Nicolaj Kamensek]

Hello list,

unfortunately, last night we experienced a total outage of our network. 
After having logged in using out-of-band access, everything was just normal:

* bgp sessions were up(all but multihop sessions)
* full-table was installed
* all interfaces were up
* there has been no reboot from any device
* memory was fine

The system has two FPCs with an etherchannel upon those(2x GigE on 2x 
FPC) + other GigE links. First thing I tried was restarting the FPCs. I 
restarted one after the other with absolutely no effect. Then, I 
switched masterchip on the SSB and back to the SSB-E16. That finally 
worked and everything was doing fine again.

This is the hardware:

Juniper M20, SSB-E16, SSB-E, dual RE, 2x FPC(non -E) and JunOS 7.6R1.10.

The SSB-E16 was installed a few weeks ago and since then the router did 
his job without any problems.

If it helps: the SSB-E16 heap usage is 23% and we currently have 8122 
arp entries installed so we shouldn't run into any memory limits here at 
all.

Right before the outage one of the AE links went down but was shortly 
after that up again. It seems to me that the link was up but the PIC did 
not forward traffic.

The complete logfile output from Nov 6th is available her: 
http://nopaste.biz/21055

The outage started at 01:37:26 with the message
RDP: Keepalive timeout for rdp.(fpc0:41985).(scb:chassis) (elapsed 5455) 
   and was over after my actions at about 2:17.

I hope someone here sees anything suspicious. Next thing we do is a 
software upgrade to JunOS 8.4R2.3 and maybe we replace the FPC0 and/or 
the SX PIC which is a member of the ae0 and went down(according to the 
Foundry switch it is connected to) shortly before the outage.

Thanks for any hints!

Best regards,
Nicolaj Kamensek

-- 
Accelerated IT Services GmbH
Schubertstrasse 10                        D-67251 Freinsheim
nk at accelerated.de                         http://www.accelerated.de/
Telefon: +49 69-25738580-3                Telefax: +49 69-25738580-4
HRB: 60665 - Amtsgericht Ludwigshafen     UstID: DE253684415
Geschäftsführende Gesellschafter:         Nicolaj Kamensek & Ole Krieger