[rbak-nsp] CGNAT performance issues

[Mariusz K. Grzeca]

Hi,

We are currently experiencing some major peformance issues with one of our SEs.

Our platform is SE1200 with 2xXCRP4 and 9 10ge-4-port cards, SEOS-12.1.1.12p15-Release. 7 BGP4 peers with a total of ~40Gbps throughput during the evening traffic peaks. Around 30k active CLIPS subscribers of which ~25k have a NAT policy attached (enhanced NAT with logging and paired mode).

Initially we had 2 line cards intended for BGP sessions only, 5 cards for CLIPS sessions and 2 cards reserved for other low throughput purposes. The first symptoms we experienced were reduced bandwidths for subscribers with 1Gbps service plans (500-600 Mbps instead of the usual 941 Mbps). The problem only affected NAT clients. A couple of weeks later BGP sessions started flapping. At first it seemed like we were hitting the 20Gbps per card limit and rising tail drop counters on the BGP cards seemed to confirm it.

So we fiddled with the wires and came out with a different setup - 7 cards with at most 2 ports connected, one for BGP peer and the other for CLIPS sessions. And it actually made the situation worse - rising tail drop counters on each card and even lower bandwidths during the evening traffic peaks and BGP sessions kept flapping. In order to prevent flapping BGP sessions we moved most of the BGP peers to another SmartEdge router and added 2 more 10ge-4-port cards for CLIPS sessions. Result - no more flapping but nothing has changed in terms of either bandwidth or tail drop counters.

I would be grateful for any suggestion as to the possible causes of this situation.


Thanks.