[c-nsp] Prove it's not the network!

[Peter Rathlev]

Hi Rick,

Often the only way to prove that the network is not to blame is by
finding what *IS* to blame. We spend countless hours working through
solutions that are totally not within our area of expertise, simply
because nobody else wants to.

The sniffer (SPAN port and a laptop) is a good way to start trying to
find out what goes wrong, but you have to dig into some very nasty
details about the specific protocols, and these are often not very well
documented. Mastering that discipline is what puts the "expert" in
"networking expert". ;-)

If all searches for a culprit fails, our only option is to turn the
tables, telling the concerned party exactly what kind of service we
deliver (bandwidth/latency/jitter/drops) and then let them figure out if
that's good enough. We usually use some ad hoc IP SLA measurings to back
this up, and are in the middle of implementing end-to-end IP SLA
measurements all over the network for everyone to look into. It usually
helps a LOT if you can give them the impression that you know exactly
what is going on in your network.

Bandwidth testing, although seldom any good indication of perceived
performance, are easily done with tools like IPerf og ttcp. Traceroutes,
maybe "mtr" (MyTraceroute) can be helpful in determining the source of
some problems, but we've had quite a few incidents where it was proven
that end users DO NOT know how to read them. Sometimes a traceroute can
do more harm than good.

We run a medium-ish enterprise MAN network by the way (government
health), so we have very few problems with external customers. It's
usually internal "customers" or external suppliers, both of which place
us in an easier political situation than external customers, if not for
anything else then because of the direction of the money flow. :-)

Regards,
Peter

On Tue, 2008-05-13 at 10:15 -0500, Rick Martin wrote:
> I know this is not really a Cisco specific question but it is
> definitely in support of Cisco hardware. 
> 
>  How do most of you folks prove that "the problem" is not the network?
> We utilize CA Spectrum and eHealth for availability and statistical
> analysis but in some instances that does not cut it. We don't typically
> have much trouble proving that a T1 is serving up 1.5 meg of bandwidth.
> Customers complain that their access is slow, we show that they are
> using all available bandwidth and eventually sell them more bandwidth
> and the problem is resolved.
> 
>  The more difficult effort is when there is plenty of available
> bandwidth and a particular application is slow (Outlook in the case I am
> involved in now). This is a very high level political official and we
> must come to a resolution. All tools we have available to us today
> indicate that there is not a problem with the network. Typical
> utilization on the T1 is about 500 to 600K peak during the day. Certain
> management continues to point the finger at the network. We have used
> Internet based speed tests that at times show less than 1.5Meg download
> speeds, I explain the variables in the Internet and the particular tool
> in use as well as local contention for the bandwidth etc to no avail,
> once they see less than 1.5 meg speed the finger points to the network.
> I still must somehow "prove" that the network is not the issue. 
> 
>  I am interested in an Internet speed test like tool to install at the
> core of our network that would provide a sustained upload or download
> test that would run for longer periods of time than a regular speed
> test. I would like to fill the pipe while graphing in Ehealth or as part
> of the selected tool to prove that the contracted bandwidth is available
> in both directions.
> 
>  Any recommendations for products would be appreciated. We are currently
> looking at SolarWinds WAN Killer and a traffic generator from Omnicore
> LanTraffic V2. I am also open to different "types" of solutions to point
> to where the problem is actually located. 
> 
> Thanks in advance for any suggestions
> 
> Rick Martin
> Network Engineer
> State of Arkansas, Department of Information Systems
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/