[j-nsp] Ex stack of 4 switchs stops routing, switching, ...

[Maarten van der Hoek]

Hi Laurent,

Had almost exactly the same this morning when I came in the office... 
All network traffic was still flowing, however DHCP packet's didn't (for
machine's which were not in the office during the weekend....laptop's /
pda's  / etc....everything 'new' to the switch)

Message log showed:

Jan  6 10:16:37  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
2, rtsm_id 47, msg type 2
Jan  6 10:16:41  swex2200vc /kernel: kmem type session using 58778K,
exceeding limit 40960K
Jan  6 10:16:42  swex2200vc /kernel: rt_pfe_veto: Memory over consumed. Op
1, rtsm_id 47, msg type 2
Jan  6 10:16:57  swex2200vc last message repeated 3 times

Stack consists of 2x EX2200-48T running Junos 13.2X50-D15.3

What ls your Junos ?
Found anything yet ?

Brgds,

Maarten


-----Oorspronkelijk bericht-----
Van: juniper-nsp [mailto:juniper-nsp-bounces at puck.nether.net] Namens Laurent
CARON
Verzonden: zondag 5 januari 2014 17:52
Aan: juniper-nsp at puck.nether.net
Onderwerp: [j-nsp] Ex stack of 4 switchs stops routing, switching, ...

Hi,

Running a chassis composed of 2 EX4200 and 2 EX4500.

One of the RE did reboot (by itself) on Dec 26th.

I managed to collect some logs:

Dec 25 12:21:41  swa eventd: sendto: Cannot allocate memory Dec 25 12:21:44
swa /kernel: rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2
Dec 25 12:21:59  swa last message repeated 3 times Dec 25 12:22:49  swa last
message repeated 10 times Dec 25 12:22:54  swa /kernel: rt_pfe_veto: Memory
over consumed. Op 1, rtsm_id 47, msg type 2 Dec 25 12:22:59  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 1, rtsm_id 47, msg type 2 ....
Jan  5 12:50:37  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
rtsm_id 0, msg type 10 Jan  5 12:50:38  swa rpd[20440]: RPD_KRT_Q_RETRIES:
Route Update: No buffer space available Jan  5 12:50:42  swa /kernel:
rt_pfe_veto: Memory over consumed. Op 8, rtsm_id ...
Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Memory over consumed. Op 8,
rtsm_id 0, msg type 10 Jan  5 15:09:17  swa /kernel: rt_pfe_veto: Possible
slowest client is pfem2. States processed - 117754359. States to be
processed - 27 Jan  5 15:09:22  swa /kernel: rt_pfe_veto: Memory over
consumed. Op 8, rtsm_id 0, msg type 10 Jan  5 15:09:22  swa /kernel:
rt_pfe_veto: Possible slowest client is pfem2. States processed - 117754359.
States to be processed - 27 Jan  5 15:09:26  swa rpd[20440]:
RPD_KRT_Q_RETRIES: Route Update: No buffer space available

Today the switch would only continue switching for a while but not route
packets anymore.

The arp table was empty.

Restarting routing process only rendered the switch unresponsive so I had to
reboot it via console port.

This switch only handles 3 dozens of LACP aggregates, a few of them are
10Gb, the others Gb, a few SVI, a few pure L2 VLANs, 100 firewall rules, no
dhcp snooping. I use OSPF on ~30 interfaces

The only "fancy" features I use are:
Graceful switchover
RSTP
LLDP
LLDP-Med
NSB
Ethernet storm control

Do any of you have a clue about it ?

Thanks

Laurent

_______________________________________________
juniper-nsp mailing list juniper-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp