[f-nsp] Determining the load on the ip-forwarding engine on a Bigiron
Kristian Larsson
kristian at juniks.net
Mon Oct 24 17:46:18 EDT 2005
On Mon, Oct 24, 2005 at 10:43:58PM +0200, Gunther Stammwitz wrote:
> Hello Brent,
>
> Thank you very much for your reply. This was a help to me.
> #show ip cache
> Total number of cache entries: 7529
> D:Dynamic P:Permanent F:Forward U:Us C:Complex Filter
> W:Wait ARP I:ICMP Deny K:Drop R:Fragment S:Snap Encap
> IP Address Next Hop MAC Type Port Vlan
> Pri
> 1 205.242.18.180 x.x.x.x 00d0.02e5.1bfc DF 2/4 491
> 0
> 2 62.96.30.11 x.x.x.x 0012.1e05.58db DF 1/1 492
> 0
> 3 213.247.32.99 DIRECT 0000.0000.0000 PU n/a
> 0
> [..]
> 1349 80.255.255.255 DIRECT 0000.0000.0000 PU n/a
> 0
> 1350 255.255.255.255 DIRECT 0000.0000.0000 PU n/a
> 0
> The ip-cache currently shows 7529 entries but when going through the full
> list it ends at entry number 1350. Interesting... Maybe the entries are
> timing out that fast? I have seen that one can set the timeout by "ip cache
> age-time / DECIMAL amount of time before idle ip cache is age out"
> but the online help doesn't list the format in which the values is given.
> Maybe seconds?
> It would also be interesting to see what the default value is. Any idea how
> to find out?
>
>
> The size of the cache itself seems to be configurable too:
> system-max ip-cache
> DECIMAL Valid range 8000 to 400000 (default: 140000)
>
>
> But I guess the interesting thing is how many entries can be kept directly
> in the cam as you say and doesn't need to be paged to the dram.
> On an older B8GMR3-A-module the size seems to be limited to 512KB per DMA.
> SL 1: B8GMR Fiber Management Module, SYSIF 2, M3, ACTIVE
> Serial #: xxxx
> 4096 KB BRAM, SMC version 1, BM version 21
> 512 KB PRAM(512K+0K) and 2048*8 CAM entries for DMA 0, version 0209
> 512 KB PRAM(512K+0K) and shared CAM entries for DMA 1, version 0209
> 512 KB PRAM(512K+0K) and 2048*8 CAM entries for DMA 2, version 0209
> 512 KB PRAM(512K+0K) and shared CAM entries for DMA 3, version 0209
>
> On a newer Jetcore module this size has been designed much larger...
> SL 1: J-Bx2GMR4 JetCore Management Module, SYSIF 2 (Mini GBIC), M4, ACTIVE
> Serial #: xxxx
> 4096 KB BRAM, JetCore ASIC IGC version 49, BIA version 8a
> 32768 KB PRAM and 2M-Bit*1 CAM for IGC 0, version 0449
>
>
> The interesting part seems to be how the cam is being divided up between
> layer2,3 and 4 and how much there is left:
> #show cam-partition brief
> ==== SLOT 1 CAM PARTITION ====
> DMA: 0 (0x00)
> Number of CAM devices per DMA: 8
> Number of hw entries per CAM: 0x00800
> Total size of CAM = 1Mbits
> complete CAM index range per DMA:
> (sw) 1 - 16383 (1 - 0x03fff), total entries: 16383 (0x03fff)
> (hw) 0 - 16383 (0 - 0x03fff), total entries: 16384 (0x04000)
> Percentage of CAM hardware entries for each partition:
> Layer3 = 12287 (0.749938Mbits) (74.993896%)
> Level3 = 2047 (0.124938Mbits) (12.493896%)
> Level2 = 2048 (0.125Mbits) (12.5%)
> Layer4 Pool0 = 2048 (0.125Mbits) (12.5%)
> Layer2/Layer4 Pool1,2,3 = 2048 (0.125Mbits) (12.5%)
> DMA: 2 (0x02)
> Number of CAM devices per DMA: 8
> Number of hw entries per CAM: 0x00800
> Total size of CAM = 1Mbits
> complete CAM index range per DMA:
> (sw) 1 - 16383 (1 - 0x03fff), total entries: 16383 (0x03fff)
> (hw) 0 - 16383 (0 - 0x03fff), total entries: 16384 (0x04000)
> Percentage of CAM hardware entries for each partition:
> Layer3 = 12287 (0.749938Mbits) (74.993896%)
> Level3 = 2047 (0.124938Mbits) (12.493896%)
> Level2 = 2048 (0.125Mbits) (12.5%)
> Layer4 Pool0 = 2048 (0.125Mbits) (12.5%)
> Layer2/Layer4 Pool1,2,3 = 2048 (0.125Mbits) (12.5%)
>
> AS one can see there are 16383 entries on the B8GMR3-A. To me it looks like
> the system is creating an ip-cache-entry for every destination-ip of the
> router. The whole Internet currently has about 170.000 entries so this is
> the theoretical maximum but I guess there is no one who sends to *all* hosts
> at the same time so this value seems to be sufficient if we keep in mind
> that there is a timeout for the ip-cache.
That's 170.000 _prefixes_. I don't think there are
smaller prefixes than /24 out there today so with
that in mind - yes you can easily fill your CAM,
that is why the net-aggregate thing was invented.
ip net-aggregate allows the CAM to save a cache
entry for a whole net instead of for a single
host.
> The number of cam entries on our router seems to be somewhere between 6000
> and 8000 over the time so we're at about 50% load but maybe someone from
> foundry can explain how the mapping between DMAs, modules and Ethernet-ports
> on those modules is being handled?
>
> Another friendly colleague told me to use the command "dm hist" which shows
> if there are any overruns of the cam and luckily there aren't any.
>
> Let's see what happens if the next blaster-like worm appears and tries to
> send traffic to a lot of different ips. Maybe one will hit the maximum cam
> entries?
I have. Actually your very email made me check our
equipment and I found a box not having ip
net-aggragate thus show ip cache:
Total number of cache entries: 140000
net-aggregate is your friend! :)
Kristian.
>
>
>
> One can also assume that game server-traffic where there's a constant
> communication between servers and a whole bunch of clients is much more
> stressing for a router since the ip-cache can't time out like with
> http-requests where there is no keep-alive, right?
>
>
>
> What I have understood so far is that as long as routing is being done via
> ip-cache and cam the cpu doesn't have to do anything with routing but my
> question is still not completely answered: what is the limit of the
> ip-forwarding-hardware? "Gigabit wire speed" can't be the answer since small
> packets are much more stressing for a router than larger ones. Any ideas?
>
> Thanks to all of you who have answered so far :-)
>
>
> Best regards,
> Gunther
>
>
>
> > -----Ursprüngliche Nachricht-----
> > Von: Brent Van Dussen [mailto:vandusb at attens.com]
> > Gesendet: Montag, 24. Oktober 2005 18:52
> > An: Gunther Stammwitz; foundry-nsp at puck.nether.net
> > Betreff: Re: [f-nsp] Determining the load on the
> > ip-forwarding engine on a Bigiron
> >
> > Hello Gunter,
> >
> > In terms of load we have always just had to keep an eye on
> > the forwarding-cache size to get a gauge for when the switch
> > is going to start having trouble.
> >
> > Since most of the forwarding is done in hardware at line-rate
> > for all prefixes when net-aggregate is turned on, there
> > usually isn't an issue. When you start to run into net-agg
> > ineligibles, this necessitates the use of the CPU on the
> > management card to keep track of and forward the non
> > conforming packets. Since the CAM isn't big enough to carry
> > all internet host's as separate entries, the CPU keeps track
> > of them in a slower DRAM location that we call the ip-cache
> > table ( 'show ip cache' ).
> >
> > So compare your current ip cache size with that of your
> > maximum configured ip-cache size and you have a rough idea
> > where you are at. Kind of an apples to Oranges comparison
> > since a bigiron works nothing like a GSR does.
> >
> > I am not sure if CPU manipulation of the ip cache gets
> > computed in with the "IP" section of show proc cpu, maybe a
> > kind foundry developer can let us know :)
> >
> > -Brent
> >
> > At 11:30 AM 10/23/2005, Gunther Stammwitz wrote:
> > >Hello colleagues,
> > >
> > >we're using several foundry Bigirons - some Ironcore and
> > some Jetcore -
> > >and I'd like to find out how much load there is on the
> > ip-forwarding-engine.
> > >On most Cisco routers one can check the cpu load of the
> > Linecard but on
> > >a foundry forwarding is done in hardware. Is there any way
> > to find out
> > >how much room there is left and how high the utilization is?
> > >
> > >As far as I understand a forwarding engine can usually
> > forward x mpps
> > >with a specific packet size which equals y megabits per
> > second but much
> > >less megabits if the packet size is smaller.
> > >
> > >Any ideas?
> > >I've seen show proc cpu and one can read
> > >"IP 0.19 0.13 0.13 0.15
> > 4518345" there
> > >but I guess this means how much cpu of the management card is being
> > >used by icmp-replies addressed to the router itself?!?
> > >
> > >Thanks,
> > >Gunther
> > >_______________________________________________
> > >foundry-nsp mailing list
> > >foundry-nsp at puck.nether.net
> > >http://puck.nether.net/mailman/listinfo/foundry-nsp
> >
> >
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
More information about the foundry-nsp
mailing list