[c-nsp] Re: FE ignored errors

[Jon Lewis]

I forgot to mention, IOS versions in use are:

rsp-k91pv-mz.122-18.S5.bin
rsp-k91pv-mz.122-18.S6.bin

On Sun, 19 Dec 2004, Jon Lewis wrote:

> I've recently started seeing large bursts (sometimes tens, usually
> thousands) of input ignored errors on several 7500 routers FE
> interfaces (rsp4, vip2-50, PA-FE-TX).  It's even happening on relatively
> low traffic ones that are only doing about 1/5 line rate traffic.
>
> I've seen Rodney's post from 11-24-04 warning not to try tuning the
> buffers, and have removed all custom hold-queue settings from the config
> on the least busy of the 7500s, and it's still having this problem.
>
> We're seeing this primarily on FE internet transit connections, but also
> to some degree on internal portchannel interfaces between our routers and
> 3550 switches that fan out to customer aggregation 3550 switches.
>
> The feature I'm guessing could be to blame is policy routing.  Ever since
> the nachi/welchia outbreak that followed blaster, I've had policy routing
> setup on all our transit connections to block the 92 byte echo/echo-reply
> packets nachi was famous for sending to pretty much the whole internet.
>
> I also have a short input ACL on all our transit interfaces that blocks
> SQL slammer and the DoS mentioned at http://www.lurhq.com/cisco-dos.html
>
> i.e. here's some config snippets from one of the routers.
>
> ip cef distributed
> interface FastEthernet6/0/0
>  ip access-group slammer+ciscodosopt in
>  ip verify unicast source reachable-via any
>  no ip unreachables
>  ip route-cache flow
>  no ip mroute-cache
>  ip policy route-map nachiworm
>  load-interval 30
>  full-duplex
>
> route-map nachiworm permit 10
>  match ip address nachilist
>  match length 92 92
>  set interface Null0
>
> ip access-list extended nachilist
>  permit icmp any any echo
>  permit icmp any any echo-reply
>
> ip access-list extended slammer+ciscodosopt
>  permit tcp any any
>  deny   udp any any eq 1434
>  permit udp any any
>  permit icmp any any
>  deny   53 any any log-input
>  deny   55 any any log-input
>  deny   77 any any log-input
>  deny   pim any any log-input
>  permit ip any any
>
> We still get quite a few hits on 1434/udp, and what could either be nachi
> or just people doing windows traceroute, and relatively few on the odd
> protos DoS.
>
> Short of just pulling all this (the policy routing and ACL) out and hoping
> our network isn't destroyed from within by infected customers, is there
> any way to diagnose whats causing the ignored errors?  I'm not seeing
> anything from show buffer fail (either on the RSP or VIPs) when we get
> bursts of ignores.  Show int stat shows what I think are normal values
> (most traffic being dcef switched, orders of magnitude less being
> processor switched).
>
> FastEthernet6/0/0
>           Switching path    Pkts In   Chars In   Pkts Out  Chars Out
>                Processor       9661    2256916      12471    1185150
>              Route cache          0          0          0          0
>        Distributed cache   34054578 10195735908   18164802 8416631117
>                    Total   34064239 10197992824   18177273 8417816267
>
> Processor load and memory (either on the RSP or VIPs) doesn't look
> problematic.
>
> ----------------------------------------------------------------------
>  Jon Lewis                   |  I route
>  Senior Network Engineer     |  therefore you are
>  Atlantic Net                |
> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>

----------------------------------------------------------------------
 Jon Lewis                   |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________