AW: [f-nsp] Determining the load on the ip-forwarding engine on a Bigiron
Gunther Stammwitz
gstammw at gmx.net
Mon Oct 24 16:43:58 EDT 2005
Hello Brent,
Thank you very much for your reply. This was a help to me.
#show ip cache
Total number of cache entries: 7529
D:Dynamic P:Permanent F:Forward U:Us C:Complex Filter
W:Wait ARP I:ICMP Deny K:Drop R:Fragment S:Snap Encap
IP Address Next Hop MAC Type Port Vlan
Pri
1 205.242.18.180 x.x.x.x 00d0.02e5.1bfc DF 2/4 491
0
2 62.96.30.11 x.x.x.x 0012.1e05.58db DF 1/1 492
0
3 213.247.32.99 DIRECT 0000.0000.0000 PU n/a
0
[..]
1349 80.255.255.255 DIRECT 0000.0000.0000 PU n/a
0
1350 255.255.255.255 DIRECT 0000.0000.0000 PU n/a
0
The ip-cache currently shows 7529 entries but when going through the full
list it ends at entry number 1350. Interesting... Maybe the entries are
timing out that fast? I have seen that one can set the timeout by "ip cache
age-time / DECIMAL amount of time before idle ip cache is age out"
but the online help doesn't list the format in which the values is given.
Maybe seconds?
It would also be interesting to see what the default value is. Any idea how
to find out?
The size of the cache itself seems to be configurable too:
system-max ip-cache
DECIMAL Valid range 8000 to 400000 (default: 140000)
But I guess the interesting thing is how many entries can be kept directly
in the cam as you say and doesn't need to be paged to the dram.
On an older B8GMR3-A-module the size seems to be limited to 512KB per DMA.
SL 1: B8GMR Fiber Management Module, SYSIF 2, M3, ACTIVE
Serial #: xxxx
4096 KB BRAM, SMC version 1, BM version 21
512 KB PRAM(512K+0K) and 2048*8 CAM entries for DMA 0, version 0209
512 KB PRAM(512K+0K) and shared CAM entries for DMA 1, version 0209
512 KB PRAM(512K+0K) and 2048*8 CAM entries for DMA 2, version 0209
512 KB PRAM(512K+0K) and shared CAM entries for DMA 3, version 0209
On a newer Jetcore module this size has been designed much larger...
SL 1: J-Bx2GMR4 JetCore Management Module, SYSIF 2 (Mini GBIC), M4, ACTIVE
Serial #: xxxx
4096 KB BRAM, JetCore ASIC IGC version 49, BIA version 8a
32768 KB PRAM and 2M-Bit*1 CAM for IGC 0, version 0449
The interesting part seems to be how the cam is being divided up between
layer2,3 and 4 and how much there is left:
#show cam-partition brief
==== SLOT 1 CAM PARTITION ====
DMA: 0 (0x00)
Number of CAM devices per DMA: 8
Number of hw entries per CAM: 0x00800
Total size of CAM = 1Mbits
complete CAM index range per DMA:
(sw) 1 - 16383 (1 - 0x03fff), total entries: 16383 (0x03fff)
(hw) 0 - 16383 (0 - 0x03fff), total entries: 16384 (0x04000)
Percentage of CAM hardware entries for each partition:
Layer3 = 12287 (0.749938Mbits) (74.993896%)
Level3 = 2047 (0.124938Mbits) (12.493896%)
Level2 = 2048 (0.125Mbits) (12.5%)
Layer4 Pool0 = 2048 (0.125Mbits) (12.5%)
Layer2/Layer4 Pool1,2,3 = 2048 (0.125Mbits) (12.5%)
DMA: 2 (0x02)
Number of CAM devices per DMA: 8
Number of hw entries per CAM: 0x00800
Total size of CAM = 1Mbits
complete CAM index range per DMA:
(sw) 1 - 16383 (1 - 0x03fff), total entries: 16383 (0x03fff)
(hw) 0 - 16383 (0 - 0x03fff), total entries: 16384 (0x04000)
Percentage of CAM hardware entries for each partition:
Layer3 = 12287 (0.749938Mbits) (74.993896%)
Level3 = 2047 (0.124938Mbits) (12.493896%)
Level2 = 2048 (0.125Mbits) (12.5%)
Layer4 Pool0 = 2048 (0.125Mbits) (12.5%)
Layer2/Layer4 Pool1,2,3 = 2048 (0.125Mbits) (12.5%)
AS one can see there are 16383 entries on the B8GMR3-A. To me it looks like
the system is creating an ip-cache-entry for every destination-ip of the
router. The whole Internet currently has about 170.000 entries so this is
the theoretical maximum but I guess there is no one who sends to *all* hosts
at the same time so this value seems to be sufficient if we keep in mind
that there is a timeout for the ip-cache.
The number of cam entries on our router seems to be somewhere between 6000
and 8000 over the time so we're at about 50% load but maybe someone from
foundry can explain how the mapping between DMAs, modules and Ethernet-ports
on those modules is being handled?
Another friendly colleague told me to use the command "dm hist" which shows
if there are any overruns of the cam and luckily there aren't any.
Let's see what happens if the next blaster-like worm appears and tries to
send traffic to a lot of different ips. Maybe one will hit the maximum cam
entries?
One can also assume that game server-traffic where there's a constant
communication between servers and a whole bunch of clients is much more
stressing for a router since the ip-cache can't time out like with
http-requests where there is no keep-alive, right?
What I have understood so far is that as long as routing is being done via
ip-cache and cam the cpu doesn't have to do anything with routing but my
question is still not completely answered: what is the limit of the
ip-forwarding-hardware? "Gigabit wire speed" can't be the answer since small
packets are much more stressing for a router than larger ones. Any ideas?
Thanks to all of you who have answered so far :-)
Best regards,
Gunther
> -----Ursprüngliche Nachricht-----
> Von: Brent Van Dussen [mailto:vandusb at attens.com]
> Gesendet: Montag, 24. Oktober 2005 18:52
> An: Gunther Stammwitz; foundry-nsp at puck.nether.net
> Betreff: Re: [f-nsp] Determining the load on the
> ip-forwarding engine on a Bigiron
>
> Hello Gunter,
>
> In terms of load we have always just had to keep an eye on
> the forwarding-cache size to get a gauge for when the switch
> is going to start having trouble.
>
> Since most of the forwarding is done in hardware at line-rate
> for all prefixes when net-aggregate is turned on, there
> usually isn't an issue. When you start to run into net-agg
> ineligibles, this necessitates the use of the CPU on the
> management card to keep track of and forward the non
> conforming packets. Since the CAM isn't big enough to carry
> all internet host's as separate entries, the CPU keeps track
> of them in a slower DRAM location that we call the ip-cache
> table ( 'show ip cache' ).
>
> So compare your current ip cache size with that of your
> maximum configured ip-cache size and you have a rough idea
> where you are at. Kind of an apples to Oranges comparison
> since a bigiron works nothing like a GSR does.
>
> I am not sure if CPU manipulation of the ip cache gets
> computed in with the "IP" section of show proc cpu, maybe a
> kind foundry developer can let us know :)
>
> -Brent
>
> At 11:30 AM 10/23/2005, Gunther Stammwitz wrote:
> >Hello colleagues,
> >
> >we're using several foundry Bigirons - some Ironcore and
> some Jetcore -
> >and I'd like to find out how much load there is on the
> ip-forwarding-engine.
> >On most Cisco routers one can check the cpu load of the
> Linecard but on
> >a foundry forwarding is done in hardware. Is there any way
> to find out
> >how much room there is left and how high the utilization is?
> >
> >As far as I understand a forwarding engine can usually
> forward x mpps
> >with a specific packet size which equals y megabits per
> second but much
> >less megabits if the packet size is smaller.
> >
> >Any ideas?
> >I've seen show proc cpu and one can read
> >"IP 0.19 0.13 0.13 0.15
> 4518345" there
> >but I guess this means how much cpu of the management card is being
> >used by icmp-replies addressed to the router itself?!?
> >
> >Thanks,
> >Gunther
> >_______________________________________________
> >foundry-nsp mailing list
> >foundry-nsp at puck.nether.net
> >http://puck.nether.net/mailman/listinfo/foundry-nsp
>
>
More information about the foundry-nsp
mailing list