[j-nsp] Ex stack of 4 switchs stops routing, switching, ...

[david.roy at orange.com]

Je vais refaire un mail. 
David


 
David Roy 
IP/MPLS NOC engineer - Orange France
Ph. : +33 2 99 87 64 72
Mob. : +33 6 85 52 22 13
SkypeID : davidroy.35
david.roy at orange.com
 
/--------------------/
JNCIE-SP #703 
JNCIE-ENT #305
JNCIP-SEC

-----Message d'origine-----
De : juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net] De la part de Maarten van der Hoek
Envoyé : lundi 6 janvier 2014 10:15
À : 'Laurent CARON'
Cc : juniper-nsp at puck.nether.net
Objet : Re: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Hi Laurent,

Had almost exactly the same this morning when I came in the office... 
All network traffic was still flowing, however DHCP packet's didn't (for
machine's which were not in the office during the weekend....laptop's /
pda's  / etc....everything 'new' to the switch)

Message log showed:

Jan  6 10:16:37  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
2, rtsm_id 47, msg type 2
Jan  6 10:16:41  swex2200vc /kernel: kmem type session using 58778K,
exceeding limit 40960K
Jan  6 10:16:42  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
1, rtsm_id 47, msg type 2
Jan  6 10:16:57  swex2200vc last message repeated 3 times

Stack consists of 2x EX2200-48T running Junos 13.2X50-D15.3

What ls your Junos ?
Found anything yet ?

Brgds,

Maarten


-----Oorspronkelijk bericht-----
Van: juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net] Namens Laurent
CARON
Verzonden: zondag 5 januari 2014 17:52
Aan: juniper-nsp at puck.nether.net
Onderwerp: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Hi,

Running a chassis composed of 2 EX4200 and 2 EX4500.

One of the RE did reboot (by itself) on Dec 26th.

I managed to collect some logs:

Dec 25 12:21:41  swa eventd: sendto: Cannot allocate memory Dec 25 12:21:44
swa /kernel: rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2
Dec 25 12:21:59  swa last message repeated 3 times Dec 25 12:22:49  swa last
message repeated 10 times Dec 25 12:22:54  swa /kernel: rt_pfe_veto: Memory
over consumed. Op 1, rtsm_id 47, msg type 2 Dec 25 12:22:59  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 ....
Jan  5 12:50:37  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
rtsm_id 0, msg type 10 Jan  5 12:50:38  swa rpd[20440]: RPD_KRT_Q_RETRIES:
Route Update: No buffer space available Jan  5 12:50:42  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 8, rtsm_id ...
Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
rtsm_id 0, msg type 10 Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Possible
slowest client is pfem2. States processed - 117754359. States to be
processed - 27 Jan  5 15:09:22  swa /kernel: rt_pfe_veto: Memory over
consumed. Op 8, rtsm_id 0, msg type 10 Jan  5 15:09:22  swa /kernel:
rt_pfe_veto: Possible slowest client is pfem2. States processed - 117754359.
States to be processed - 27 Jan  5 15:09:26  swa rpd[20440]:
RPD_KRT_Q_RETRIES: Route Update: No buffer space available

Today the switch would only continue switching for a while but not route
packets anymore.

The arp table was empty.

Restarting routing process only rendered the switch unresponsive so I had to
reboot it via console port.

This switch only handles 3 dozens of LACP aggregates, a few of them are
10Gb, the others Gb, a few SVI, a few pure L2 VLANs, 100 firewall rules, no
dhcp snooping. I use OSPF on ~30 interfaces

The only "fancy" features I use are:
Graceful switchover
RSTP
LLDP
LLDP-Med
NSB
Ethernet storm control

Do any of you have a clue about it ?

Thanks

Laurent

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.