[c-nsp] Re: FE ignored errors

Sun Dec 19 20:22:59 EST 2004

Jon

What other PA exists on this VIP ? And what other VIP's (any VIP2-40's
?) Also, can you check the CpU util on the VIP itself ? Note that the
CPU utilisation has lesser bearing on router performance but may impact
latency, and it becomes a more linear curve when a per-packet processing
services exist. 

For your nachi filters, I would rather replace it with a CAR with a rate
limit of 128K or something sensible (depending on your external links)
like that policing ICMP traffic. Hence eliminating per-packet processing
done by POLICY ROUTING.

We had a similar issue with a VIP4-80 on a 7500, which had a
PA-A3-OC3-SMI & PA-FE-TX.
The ATM PA had a lot of PVC's to the remote sites, and the PA-FE-TX
connected to the backbone network. 
After extensive investigation, the problem was found to be related to
the oversubscription of traffic exiting the router on exit VIP (which
was a 2-40) - causing a bottleneck on the entry VIP (a VIP4-80).

Rgds
Nick

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Jon Lewis
Sent: Monday, 20 December 2004 10:51 AM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] Re: FE ignored errors

I forgot to mention, IOS versions in use are:

rsp-k91pv-mz.122-18.S5.bin
rsp-k91pv-mz.122-18.S6.bin

On Sun, 19 Dec 2004, Jon Lewis wrote:

> I've recently started seeing large bursts (sometimes tens, usually
> thousands) of input ignored errors on several 7500 routers FE
> interfaces (rsp4, vip2-50, PA-FE-TX).  It's even happening on 
> relatively low traffic ones that are only doing about 1/5 line rate 
> traffic.
>
> I've seen Rodney's post from 11-24-04 warning not to try tuning the
> buffers, and have removed all custom hold-queue settings from the 
> config on the least busy of the 7500s, and it's still having this 
> problem.
>
> We're seeing this primarily on FE internet transit connections, but
> also to some degree on internal portchannel interfaces between our 
> routers and 3550 switches that fan out to customer aggregation 3550 
> switches.
>
> The feature I'm guessing could be to blame is policy routing.  Ever
> since the nachi/welchia outbreak that followed blaster, I've had 
> policy routing setup on all our transit connections to block the 92 
> byte echo/echo-reply packets nachi was famous for sending to pretty 
> much the whole internet.
>
> I also have a short input ACL on all our transit interfaces that
> blocks SQL slammer and the DoS mentioned at 
> http://www.lurhq.com/cisco-dos.html
>
> i.e. here's some config snippets from one of the routers.
>
> ip cef distributed
> interface FastEthernet6/0/0
>  ip access-group slammer+ciscodosopt in
>  ip verify unicast source reachable-via any
>  no ip unreachables
>  ip route-cache flow
>  no ip mroute-cache
>  ip policy route-map nachiworm
>  load-interval 30
>  full-duplex
>
> route-map nachiworm permit 10
>  match ip address nachilist
>  match length 92 92
>  set interface Null0
>
> ip access-list extended nachilist
>  permit icmp any any echo
>  permit icmp any any echo-reply
>
> ip access-list extended slammer+ciscodosopt
>  permit tcp any any
>  deny   udp any any eq 1434
>  permit udp any any
>  permit icmp any any
>  deny   53 any any log-input
>  deny   55 any any log-input
>  deny   77 any any log-input
>  deny   pim any any log-input
>  permit ip any any
>
> We still get quite a few hits on 1434/udp, and what could either be
> nachi or just people doing windows traceroute, and relatively few on 
> the odd protos DoS.
>
> Short of just pulling all this (the policy routing and ACL) out and
> hoping our network isn't destroyed from within by infected customers, 
> is there any way to diagnose whats causing the ignored errors?  I'm 
> not seeing anything from show buffer fail (either on the RSP or VIPs) 
> when we get bursts of ignores.  Show int stat shows what I think are 
> normal values (most traffic being dcef switched, orders of magnitude 
> less being processor switched).
>
> FastEthernet6/0/0
>           Switching path    Pkts In   Chars In   Pkts Out  Chars Out
>                Processor       9661    2256916      12471    1185150
>              Route cache          0          0          0          0
>        Distributed cache   34054578 10195735908   18164802 8416631117
>                    Total   34064239 10197992824   18177273 8417816267
>
> Processor load and memory (either on the RSP or VIPs) doesn't look
> problematic.
>
> ----------------------------------------------------------------------
>  Jon Lewis                   |  I route
>  Senior Network Engineer     |  therefore you are
>  Atlantic Net                |
> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>

----------------------------------------------------------------------
 Jon Lewis                   |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

------------------------------------------------------------------------------
This communication, including any attachments, is confidential. If 
 you are not the intended recipient, you should not read it - please 
 contact me immediately, destroy it, and do not copy or use any part of 
 this communication or disclose anything about it.

------------------------------------------------------------------------------