[c-nsp] Sup2T netflow problems

[Henri Grönroos]

On 5.2.2014 13:47, Peter Rathlev wrote:
> We've started seeing some problems with our netflow collection and
> export from Sup2T's running 15.1(1)SY AIS.
>
> The problems started when we suddenly didn't see any flows exported from
> the device in question. Trying to show the flow cache from CLI just
> makes that VTY hang (can't be cleared):
>
>    Sup2T#show platform flow ip source 10.0.2.1
>    [hangs forever...]
>
> The result is the same with the "show flow" way:
>
>    Sup2T#show flow monitor STANDARD-INGRESS-IPV4 cache
>    [hangs forever...]
>
> (Maybe of interest here: The VTY lines have "exec prompt timestamp"
> configured, but the prompt hangs before the timestamp is shown.)
>
> We cannot clear the VTY sessions; we tried "clear line vty X", "clear
> line Y" and settings "exec-timeout 1 0" on the line, all to no avail.
> The TCP sessions are closed correctly when we forcibly close the SSH
> session.
>
> Trying to remove all Netflow configuration doesn't succeed. We can
> remove all the monitors from interfaces (134 of them at the moment; the
> last interface to have a monitor removed takes _very_ long time by the
> way, but that's more of a nuisance) but cannot delete the flow monitor
> afterwards:
>
>    ...
>    Sup2T(config)#no flow monitor STANDARD-INGRESS-IPV4
>    Sup2T(config)#no flow exporter STANDARD-NDE
>    % Flow Exporter: Flow Exporter STANDARD-NDE is in use. Remove from all clients before deleting.
>    Sup2T(config)#no flow record IPV4-FULL
>    % Flow Record: Flow Record is in use. Remove from all clients before deleting.
>    Sup2T(config)#no flow monitor STANDARD-INGRESS-IPV4
>    Sup2T(config)#flow monitor STANDARD-INGRESS-IPV4
>    % Flow Monitor: could not create monitor.
>    Sup2T(config)#
>
> It still appears in the configuration:
>
>    Sup2T#show running-config partition common | section ^flow
>    flow record IPV4-FULL
>     match ipv4 tos
>     match ipv4 protocol
>     match ipv4 source address
>     match ipv4 destination address
>     match transport source-port
>     match transport destination-port
>     collect transport tcp flags
>     collect interface input
>     collect counter bytes long
>     collect counter packets long
>     collect timestamp sys-uptime first
>     collect timestamp sys-uptime last
>    flow exporter STANDARD-NDE
>     destination 192.0.2.10
>     source Loopback0
>     transport udp 30002
>    flow platform cache timeout inactive 120
>    flow platform cache timeout active 300
>    flow monitor STANDARD-INGRESS-IPV4
>     exporter STANDARD-NDE
>     record IPV4-FULL
>    flow hardware usage notify input 80 1800
>    Sup2T#
>
> But not in the auto-complete list from exec mode:
>
>    Sup2T#show flow monitor ?
>      broker  Show the flow monitor broker
>      type    Type of the Flow Monitor
>      No monitors available              <----
>      |       Output modifiers
>      <cr>
>
> Typing it manually doesn't help:
>
>    Sup2T#show flow monitor STANDARD-INGRESS-IPV4 cache
>                            ^
>    % Invalid input detected at '^' marker.
>
> We're guessing a reload of the box would help (though the hanging VTY
> lines may mean we have to cut power) but would like for this to not
> happen again.
>
> The box is running 15.1(1)SY (s2t54-advipservicesk9-mz.SPA.151-1.SY.bin)
> currently and we have a planned upgrade in the near future to 15.1(2)SY1
> (s2t54-advipservicesk9-mz.SPA.151-2.SY1.bin).
>
> I found a possibly relevant thread here:
>
> https://supportforums.cisco.com/thread/2237229
>
> We'll try contacting our Cisco partner, but maybe someone here has seen
> the problem before and knows of either a work-around or that it is fixed
> in some newer software version.
>
> TIA.
>

Hi Peter,

I think you are encountering CSCui17732 which is present in 15.1.2-SY1 too.

"Sup2T: show tech-support hangs VTY session on Netflow TCAM interrupt"

In our Sup2Ts when that occurs they print syslog message
%EARL_L3_ASIC-3-INTR_FATAL: EARL L3 ASIC 0: fatal interrupt NF_SE_CMD_ERR.

After that Sup2Ts do export flows from traffic that hits control plane, 
but hardware
export is broken.

regards,
-- 
Henri Grönroos