[c-nsp] A9K Netflow export drops

[Robert Williams]

Hi,

I've got an issue on one of our smaller 9001 boxes which is puzzling me.

It suffers from a high rate of netflow export drops (not cache drops) shown here:

RP/0/RSP0/CPU0:#show flow exporter xxx location 0/0/CPU0
Sat May 21 11:44:33.008 BST
Flow Exporter: xxx
Flow Exporter memory usage: 3281158
Used by flow monitors: ALL
                       ALL_V6

Status: Normal
Transport:   UDP
Destination: x.x.x.x   (5121) VRF default
Source:      y.y.y.y   (21159)
Flows exported:                             1910954 (120244810 bytes)
Flows dropped:                                64740 (7056660 bytes)       <<<<<<<  lots of drops :(

Templates exported:                            1931 (193852 bytes)
Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes)
Option data dropped:                              0 (0 bytes)

Option templates exported:                        0 (0 bytes)
Option templates dropped:                         0 (0 bytes)

Packets exported:                             91612 (248360686 bytes)
Packets dropped:                               4980 (7091520 bytes)

Total export over last interval of:
  1 hour:                                     86814 pkts
                                          114700404 bytes
                                            1749217 flows
  1 minute:                                    5747 pkts        <<<<<<  ~6k pps rate, what's the limit?
                                            7618148 bytes
                                             116209 flows
  1 second:                                       4 pkts
                                               4324 bytes
                                                 60 flows


RP/0/RSP0/CPU0:#show flow monitor ALL cache internal location 0/0/CPU0
Sat May 21 11:45:49.698 BST
Cache summary for Flow Monitor :
Cache size:                        1000000
Current entries:                     30439    <<<<<<  only 3% usage (sampler @ 1:20)
Flows added:                       1257008
Flows not added:                         0    <<<<<<  all good
Ager Polls:                            633
  - Active timeout                   25329
  - Inactive timeout               1075287
  - TCP FIN flag                    125953
  - Emergency aged                       0
  - Counter wrap aged                    0
  - Total                          1226569
Periodic export:
  - Counter wrap                         0
  - TCP FIN flag                         0
Flows exported                     1226569


So from what I understand, it is capturing the flows OK but is unable to get the flow data out, for some reason.

The relevant config snippets are:

interface <several TE ports>
 flow ipv4 monitor ALL sampler SAMPLER ingress

flow exporter-map xxx
 version v9
  template data timeout 30
 !
 transport udp 5121
 source Loopback0
 destination x.x.x.x

flow monitor-map ALL
 record ipv4
 exporter xxx
 cache entries 1000000
 cache timeout active 60
 cache timeout inactive 10
 cache timeout rate-limit 1000000

sampler-map SAMPLER
 random 1 out-of 20

So - what am I missing here? Surely with a cache capability of 1M it should be ok to export flows when were are only around 30,000 of them nicely ticking over?

(Note that the timeouts are deliberately low because we use this data as a backup means of inbound DDoS alerting and it needs to export active streams within 60 seconds. This is in case our primary detection system has an issue and we need data from an alternative source.)

Cheers!


Robert Williams
Custodian Data Centre
Email: Robert at CustodianDC.com
http://www.CustodianDC.com