[j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Mon Jan 6 05:28:36 EST 2014

Oups ! a little mistake. 

Sorry for the spam. 

-----Message d'origine-----
De : juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net] De la part de david.roy at orange.com
Envoyé : lundi 6 janvier 2014 11:21
À : 'Maarten van der Hoek'; 'Laurent CARON'
Cc : juniper-nsp at puck.nether.net
Objet : Re: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Je vais refaire un mail. 
David

David Roy
IP/MPLS NOC engineer - Orange France
Ph. : +33 2 99 87 64 72
Mob. : +33 6 85 52 22 13
SkypeID : davidroy.35
david.roy at orange.com

/--------------------/
JNCIE-SP #703
JNCIE-ENT #305
JNCIP-SEC

-----Message d'origine-----
De : juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net] De la part de Maarten van der Hoek Envoyé : lundi 6 janvier 2014 10:15 À : 'Laurent CARON'
Cc : juniper-nsp at puck.nether.net
Objet : Re: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Hi Laurent,

Had almost exactly the same this morning when I came in the office... 
All network traffic was still flowing, however DHCP packet's didn't (for machine's which were not in the office during the weekend....laptop's / pda's  / etc....everything 'new' to the switch)

Message log showed:

Jan  6 10:16:37  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op 2, rtsm_id 47, msg type 2 Jan  6 10:16:41  swex2200vc /kernel: kmem type session using 58778K, exceeding limit 40960K Jan  6 10:16:42  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 Jan  6 10:16:57  swex2200vc last message repeated 3 times

Stack consists of 2x EX2200-48T running Junos 13.2X50-D15.3

What ls your Junos ?
Found anything yet ?

Brgds,

Maarten

-----Oorspronkelijk bericht-----
Van: juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net] Namens Laurent CARON
Verzonden: zondag 5 januari 2014 17:52
Aan: juniper-nsp at puck.nether.net
Onderwerp: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Hi,

Running a chassis composed of 2 EX4200 and 2 EX4500.

One of the RE did reboot (by itself) on Dec 26th.

I managed to collect some logs:

Dec 25 12:21:41  swa eventd: sendto: Cannot allocate memory Dec 25 12:21:44 swa /kernel: rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 Dec 25 12:21:59  swa last message repeated 3 times Dec 25 12:22:49  swa last message repeated 10 times Dec 25 12:22:54  swa /kernel: rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 Dec 25 12:22:59  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 ....
Jan  5 12:50:37  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8, rtsm_id 0, msg type 10 Jan  5 12:50:38  swa rpd[20440]: RPD_KRT_Q_RETRIES:
Route Update: No buffer space available Jan  5 12:50:42  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 8, rtsm_id ...
Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8, rtsm_id 0, msg type 10 Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Possible slowest client is pfem2. States processed - 117754359. States to be processed - 27 Jan  5 15:09:22  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8, rtsm_id 0, msg type 10 Jan  5 15:09:22  swa /kernel:
rt_pfe_veto: Possible slowest client is pfem2. States processed - 117754359.
States to be processed - 27 Jan  5 15:09:26  swa rpd[20440]:
RPD_KRT_Q_RETRIES: Route Update: No buffer space available

Today the switch would only continue switching for a while but not route packets anymore.

The arp table was empty.

Restarting routing process only rendered the switch unresponsive so I had to reboot it via console port.

This switch only handles 3 dozens of LACP aggregates, a few of them are 10Gb, the others Gb, a few SVI, a few pure L2 VLANs, 100 firewall rules, no dhcp snooping. I use OSPF on ~30 interfaces

The only "fancy" features I use are:
Graceful switchover
RSTP
LLDP
LLDP-Med
NSB
Ethernet storm control

Do any of you have a clue about it ?

Thanks

Laurent

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.