[c-nsp] RE: 3750G ICMP echo/ICMP time exceeded treatment (was RE: [c-nsp]3750Gpacket loss)

Thu Oct 20 05:19:50 EDT 2005

Apologies for replying so quickly to my own post. I realised after I posted
that mtr was just my M$ origins showing through, be it nice and graphical.
Using the normal ping command to achieve the same results gives an
interesting overview.

Eg: expected behaviour:
krypton:~# ping -t 1 [routed IP]
PING [routed IP] ([routed IP]) 56(84) bytes of data.
>From [cisco3750g] icmp_seq=1 Time to live exceeded
>From [cisco3750g] icmp_seq=2 Time to live exceeded
>From [cisco3750g] icmp_seq=3 Time to live exceeded

However, when you increase the rate these are generated, you can see some
clear kind of throttling occurring. Below there are clumps of error
messages; I assume that the 3750 is sending a prescribed number of errors
before stopping, then its counters reset and it is allowed to send the
errors again.

krypton:~# ping -c 1000 -f -t 1 [routed IP]
PING [routed IP] ([routed IP]) 56(84) bytes of data.
.EEEEEEEEEEEEEEE...................................................EEEEEEEEE
EEEEEEEEEEEE................................................................
......EEEEEEEEEEEEEEEEEEEEE.................................................
...................EEEEEEEEEEEE.EEEEEEEE....................................
................................EEEEEEEEEEEEEEEEEEEEE.......................
.............................................EEEEEEEEEEEEEEEEEEEEE..........
...............................................................EEEEEEEEEEEEE
EEEEEEEE....................................................................
.....EEEEEEEEEEEEEEEEEEEE...................................................
.................EEEEEEEEEEEEEEEEEEEEE......................................
..................................EEEEEEEEEEEEEEEEEEEEE.....................
.................................................EEEEEEEEEEEEEEEEEEEEE......
..................................................................EEEEEEEEEE
EEEEEEEEEEE.
--- [routed IP] ping statistics ---
1000 packets transmitted, 0 received, +247 errors, 100% packet loss, time
10821ms
, pipe 2, ipg/ewma 10.832/0.000 ms 

I have reached a figure of 21pps as what a 3750G will produce of ICMP type
11 messages.

Regards,

Ed Butler
RapidSwitch Ltd
DDI: 020 7106 0731

RapidSwitch Ltd, 5th Floor, Sovereign House, 227 Marsh Wall, London, E14 9SD

This email message is intended only for the addressee(s) and contains
information that may be confidential and/or copyright.  If you are not the
intended recipient please notify the sender by reply email and immediately
delete this email. Use, disclosure or reproduction of this email by anyone
other than the intended recipient(s) is strictly prohibited. No
representation is made that this email or any attachments are free of
viruses. Virus scanning is recommended and is the responsibility of the
recipient. 
-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Ed Butler -
RapidSwitch
Sent: 20 October 2005 10:09
To: cisco-nsp at puck.nether.net
Subject: RE: 3750G ICMP echo/ICMP time exceeded treatment (was RE:
[c-nsp]3750Gpacket loss)

Hi Shivas,

I've been using mtr and ping from a Debian Linux (stable) box on a Dell
PowerEdge 1850. This was plugged straight into the 3750G.

You can download mtr from: (I just did "apt-get install mtr" and used the
Debian package) http://www.bitwizard.nl/mtr/

The system is then setup in this fashion:
Dell 1850 Linux Box -> Cisco 3750G -> Cisco 6500/sup720

I run mtr to the sup720 box:
"time mtr -i 0.0001 --report-cycles 1000 [sup720 IP]"
This times how long it takes to send 1000 ICMP packets with a TTL that has
expired, eg that will make the 3750G send an ICMP time exceeded message.
Just now I got:
9.2s to send 1000 messages, 79.8% loss on the 3750G hop.

I then run a simple echo test to the 3750G:
"time ping -f -c 1000 [c3750g IP]"
This responds to 1000 ICMP echos in 0.5s, and doesn't drop a single packet.

Can anyone else replicate these results in their labs?

Regards,

Ed Butler
RapidSwitch Ltd
DDI: 020 7106 0731

RapidSwitch Ltd, 5th Floor, Sovereign House, 227 Marsh Wall, London, E14 9SD

This email message is intended only for the addressee(s) and contains
information that may be confidential and/or copyright.  If you are not the
intended recipient please notify the sender by reply email and immediately
delete this email. Use, disclosure or reproduction of this email by anyone
other than the intended recipient(s) is strictly prohibited. No
representation is made that this email or any attachments are free of
viruses. Virus scanning is recommended and is the responsibility of the
recipient. 
-----Original Message-----
From: Sivaramakrishnan, Ts [mailto:ts.sivaramakrishnan at hp.com]
Sent: 20 October 2005 08:44
To: Ed Butler - RapidSwitch
Subject: RE: 3750G ICMP echo/ICMP time exceeded treatment (was RE: [c-nsp]
3750Gpacket loss)

 Hi,

Could you please share with how can we do this test?.

Regards
Shivas

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Ed Butler -
RapidSwitch
Sent: Wednesday, October 19, 2005 11:16 PM
To: cisco-nsp at puck.nether.net
Subject: 3750G ICMP echo/ICMP time exceeded treatment (was RE: [c-nsp]
3750Gpacket loss)

I've done some further digging on the packetloss to the 3750 stack we have
here. 

My tests show 2.5kpps of ICMP ECHO causes the CPU load to increase to 60%.
However, more concerning is the ICMP time exceeded message (eg from a
traceroute) seems not to be answered at what I'd consider a reasonable rate.

I've had as low as 60pps of type 11 ICMP messages (the time exceeded
ones) causing some loss. However, it has no discernible affect on the CPU
usage.

Is this by accident or design? Why would Cisco treat the two different types
of ICMP so differently?

Regards,

Ed Butler
RapidSwitch Ltd
DDI: 020 7106 0731

RapidSwitch Ltd, 5th Floor, Sovereign House, 227 Marsh Wall, London, E14 9SD

This email message is intended only for the addressee(s) and contains
information that may be confidential and/or copyright.  If you are not the
intended recipient please notify the sender by reply email and immediately
delete this email. Use, disclosure or reproduction of this email by anyone
other than the intended recipient(s) is strictly prohibited. No
representation is made that this email or any attachments are free of
viruses. Virus scanning is recommended and is the responsibility of the
recipient. 
-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Ed Butler -
RapidSwitch
Sent: 14 October 2005 19:28
To: cisco-nsp at puck.nether.net
Subject: RE: [c-nsp] 3750G packet loss

I agree that it's sensible for the control plane to be protected. But ICMP
is a useful problem finding tool, and it's being too aggressive for our
needs at the moment.

If the 3750 is rate limiting ICMP; are there any ways to configure this?
I'd sleep much better if I knew at what point it was rate limiting, what it
was limiting too, and had configurable limits.

We're forwarding everythiing in hardware on the 3750 stack; it's all CEF (or
dCEF to be precise). CPU usage generally hovers around 10%.

I can't find any Cisco docs on the 3750 rate-limiting ICMP, can anyone point
me to these?

Regards,

Ed Butler
RapidSwitch Ltd
DDI: 020 7106 0731

RapidSwitch Ltd, 5th Floor, Sovereign House, 227 Marsh Wall, London, E14 9SD

This email message is intended only for the addressee(s) and contains
information that may be confidential and/or copyright.  If you are not the
intended recipient please notify the sender by reply email and immediately
delete this email. Use, disclosure or reproduction of this email by anyone
other than the intended recipient(s) is strictly prohibited. No
representation is made that this email or any attachments are free of
viruses. Virus scanning is recommended and is the responsibility of the
recipient. 
-----Original Message-----
From: sthaug at nethelp.no [mailto:sthaug at nethelp.no]
Sent: 14 October 2005 18:53
To: ed.butler at rapidswitch.com
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] 3750G packet loss

> I've investigated the 3750G problem with it dropping packets to its IP

> interface as below.

Why do you believe it's a problem? A good router these days *needs* to
protect itself, which (among others) usually means some form of rate
limiting or policing of traffic to the router itself.

Steinar Haug, Nethelp consulting, sthaug at nethelp.no

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/