[c-nsp] Sampled netflow on 6500/7600
Ian Dickinson
iand at eng.pipex.net
Sun Jul 2 18:08:41 EDT 2006
Hi,
> Frankly, sampling was a retrofit in this forwarding engine architecture
> - the h/w was not designed to do "true" packet-based sampling, so the
> implementation is a close software-driven approximation.
That's a shame - like mac-accounting not being there - there's no
approximation close enough for our needs yet.
We often hear about the development constraints being set by large customers
back in the day. They seem to have a lot of money and yet low expectations
sometimes :-( Anyway, I'll remove my bitterness from this thread and go back
to facts.
>> > The only ways to scale NF with many flows today are:
>> > - more agressive aging
>>
>> Please can you advise actual config of what you mean by aggressive?
>
> Lower the normal aging timer, turn on fast aging.
Ok so I use:
mls flow ip destination
mls aging fast time 4 threshold 2
mls aging long 900
mls aging normal 32
Is this sensible? This just to try and make it calm down...
What's the expected hit on CPU?
>> > - add DFCs
>>
>> Already done - that's linear scaling - we need something much more
>> than that.
>> A 6704 or 6748 which is running ~3Gbps is already overwhelming the
>> TCAM/ICAM.
>
> BXL gives you 256K entries per slot vs A/B.
Already talking about WS-X6704-10GE/WS-X6748-SFP both with WS-F6700-DFC3BXL
Forwarding engine load:
Module pps peak-pps peak-time
1 467682 837087 23:35:52 GMT Tue May 2 2006
3 583951 951987 18:53:39 GMT Mon Jun 26 2006
Netflow Resources
TCAM utilization: Module Created Failed %Used
1 262027 0 100%
3 262034 0 100%
ICAM utilization: Module Created Failed %Used
1 12 1046243 9%
3 18 949114 14%
Switch Fabric Resources
Bus utilization: current: 0%, peak was 0% at 22:03:51 GMT Sun Jul 2 2006
Fabric utilization: Ingress Egress
Module Chanl Speed rate peak rate peak
1 0 20G 7% 11% @19:01 05May06 4% 11% @15:25 25May06
1 1 20G 4% 9% @15:42 21Jun06 7% 11% @22:15 13Jun06
3 0 20G 4% 8% @17:12 08May06 6% 12% @18:39 16May06
3 1 20G 6% 13% @12:34 25May06 4% 9% @22:00 13Jun06
Ian
> Tim
>
>
>> > However, both of these will increase the CPU utilization as you try
>> > to age/export all those flows, so there is a tradeoff and you may or
>> > may not be able to find a happy medium in your network.
>>
>> Mmm. Given that a MSFC3 is hardly super-fast that's slightly worrying.
>
>
>> > Per-interface NF is on the roadmap, which will *only* enable NF entry
>> > creation for the specified interfaces rather than all interfaces as
>> > it is today.
>>
>> I can see this helping in some scenarios, but in our case we probably
>> need 70%
>> of interfaces, 50% of the pps - again linear.
>
>
>> Ian
>>
>> > Tim
>> >
>> > At 07:28 PM 6/30/2006 -0400, Matt Stockdale uttered:
>> >
>> >>Hmm, I'm seeing a non-trival tcam load on even less traffic, but I have
>> >>a non 3bxl 720
>> >>
>> >>Summary of Netflow CAM Utilization (as a percentage)
>> >>====================================================
>> >>TCAM Utilization : 26%
>> >>ICAM Utilization : 0%
>> >>
>> >>Of course, I'm doing v5 peer-as export, but it's only on a few hundred
>> >>Mbps of traffic. (edge router)
>> >>
>> >>I guess that's not very helpful, but maybe it can confirm your
>> suspicion
>> >>that you'll need to move to sampled netflow.
>> >>
>> >>-----Original Message-----
>> >>From: cisco-nsp-bounces at puck.nether.net
>> >>[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Richard A
>> >>Steenbergen
>> >>Sent: Friday, June 30, 2006 6:39 PM
>> >>To: Bill Nash
>> >>Cc: cisco-nsp at puck.nether.net
>> >>Subject: Re: [c-nsp] Sampled netflow on 6500/7600
>> >>
>> >>On Fri, Jun 30, 2006 at 06:24:03PM -0400, Bill Nash wrote:
>> >>
>> >>>I'm not going to even pretend to have your level of expertise here,
>> >>>but I'm only seeing one or two percent tcam utilization on a
>> >>>moderately loaded 6509. I suppose it's also possible that even though
>> >>>I'm configured in such a manner that I'm still pulling 15 to 16 gigs
>> >>>of raw flows out of my network on a daily basis, I'm still doing it
>> >>
>> >>wrong.
>> >>
>> >>Well just to clarify, by moderate load I mean something like:
>> >>
>> >> Forwarding engine load:
>> >> Module pps peak-pps
>> >>peak-time
>> >> 5 1489010 2542352 13:05:48 EDT Sun May
>> >>21 2006
>> >>
>> >>show fabric utilization all:
>> >> slot channel speed Ingress % Egress %
>> >> 1 0 20G 17 6
>> >> 1 1 20G 15 12
>> >> 4 0 20G 5 10
>> >> 4 1 20G 8 16
>> >> 5 0 20G 0 0
>> >>
>> >>Aka nowhere close to "large volumes of traffic", but not completely
>> >>empty, just a typical aggregation box pushing typical internet traffic.
>> >>
>> >>Summary of Netflow CAM Utilization (as a percentage)
>> >>====================================================
>> >>TCAM Utilization : 72%
>> >>ICAM Utilization : 0%
>> >>
>> >>Destination flowmask only, v4 sampling only, v5 export, etc.
>> >>
>> >>--
>> >>Richard A Steenbergen <ras at e-gerbil.net>
>> >>http://www.e-gerbil.net/ras
>> >>GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1
>> >>2CBC) _______________________________________________
>> >>cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >>https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >>archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >>
>> >>
>> >>_______________________________________________
>> >>cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >>https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >>archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >
>> >
>> >
>> >
>> > Tim Stevenson, tstevens at cisco.com
>> > Routing & Switching CCIE #5561
>> > Technical Marketing Engineer, Catalyst 6500
>> > Cisco Systems, http://www.cisco.com
>> > IP Phone: 408-526-6759
>> > ********************************************************
>> > The contents of this message may be *Cisco Confidential*
>> > and are intended for the specified recipients only.
>> > _______________________________________________
>> > cisco-nsp mailing list cisco-nsp at puck.nether.net
>> > https://puck.nether.net/mailman/listinfo/cisco-nsp
>> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>> u
>>
>> --
>> Ian Dickinson
>> Development Engineer
>> Pipex
>> ian.dickinson at pipex.net
>> http://www.pipex.net
>>
>> This e-mail is subject to: http://www.pipex.net/disclaimer.html
>
>
>
>
> Tim Stevenson, tstevens at cisco.com
> Routing & Switching CCIE #5561
> Technical Marketing Engineer, Catalyst 6500
> Cisco Systems, http://www.cisco.com
> IP Phone: 408-526-6759
> ********************************************************
> The contents of this message may be *Cisco Confidential*
> and are intended for the specified recipients only.
--
Ian Dickinson INOC-DBA: 5413*426
Development Engineer Mobile: +44 7967 463023
Pipex Direct: +44 1865 381522
iand at eng.pipex.net (Work) Fax: +44 1865 778160
ian.dickinson at pipex.net (Corporate) http://www.pipex.net
PGP Fingerprint: 1A5E 74B1 2BDD 214A 2131 69E9 C3B3 B72A DDF8 862A
This e-mail is subject to: http://www.pipex.net/disclaimer.html
More information about the cisco-nsp
mailing list