[c-nsp] packet loss between adjacent ciscos

[Rodney Dunn]

Upgrade the 72xx's to 12.4(20)T latest on Cisco.com to get the
packet capture feature and prove where the packets are getting
lost via a capture:

http://supportwiki.cisco.com/ViewWiki/index.php/Tech_Insights:Utilizing_the_New_Packet_Capture_Feature

We could go in to the long discussion about how to make sure all the
packets are being CEF switched, performance with features, etc...

But it's easier to just prove via a trace where the packets are not being
delivered via a trace and focus there.

Rodney

On Fri, Feb 27, 2009 at 04:28:30PM +1100, Sam Tilders wrote:
> Hi,
> 
> We have been experiencing some packet loss between a switch and a
> router directly connected to each other and are having some difficulty
> finding the problem.
> 
> The problem showed up when a customer complained that there were
> moments of silence on their voip calls. They did some pings and found
> that there was packet loss at the same time as the silence on the calls.
> 
> With some further help from the customer, I was able to narrow the
> problem down to loss between two ciscos in our rack.
> 
> The network layout is like this:
> 
> carrier peer port
>       |
>       |
>       | 100% ping success.
>       |
>       | iofe
> border router (7200vxr w/npe-400 12.2(18)S4)
>       | pa-fe-tx
>       |
>       | 99.994 - 99.999% ping success
>       |
>       |
> switch (2924 xl en)
>       |
>       |
>       | 100% ping success
>       |
>       | iofe
> l2tp termination router (7200vxr w/npe-300 12.4(4)T1)
>       | gige
>       |
>       |
> downstream to customers
> 
> The ping percentages are from repeated 100000 ping samples.
> 
> The interfaces are all forced duplex full, the switch interfaces are
> forced speed 100.
> 
> When the link between the router and the switch has loss it can be
> seen in the ping as a slow down then a single timeout.
> 
> The ping output goes something like:
> 
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!! !  !   !    !     !      !         .!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> 
> where I've used spacing to indicate the time between markers.
> 
> So it appears that the ping and reply begins to slow, then it slows
> enough that a single 2 second timeout occurs and then it picks up
> again at full speed.
> 
> A 100000 ping takes a few minutes to run and during this time it may
> lose one or half a dozen pings, each lost ping spaced apart,
> apparently with no regular period.
> 
> The router and switch are typically running around 30% cpu.
> 
> When I run these ping tests the switch gets to 80% cpu, however it can
> be shown with cases like a customer's voip call that the problem
> occurs even when the util is lower.
> 
> I have correlated the ping loss with the customer's voip silence,
> having them on a call while running the ping and they experience a
> couple of seconds of silence at the same time as the router misses a
> ping.
> 
> I've been on site and replaced the pa-fe-tx in the 7200 to no
> improvement. I moved the PA to a different port on the router, no
> improvement. I've replaced the switch with no improvement.
> 
> (We had previously tried different switch ports and replacing the cabling.)
> 
> All the while, none of the interface statistics report any errors
> other than the occasional ignored packet - however, these don't occur  
> at the same time as
> the problem and much less frequently.
> 
> I've had various debug options turned on - both on the switch and the
> router - there has been no clear correlation between any events and
> the occurence of packet loss.
> 
> So, I was wondering if this sounds familiar to anyone or if there is
> anything someone might be able to suggest to further investigate or
> resolve this issue.
> 
> I'd appreciate any advice that can be given.
> 
> Regards,
> 
> - Sam
> 
> 
> 
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/