[j-nsp] M10 maximum throughput

[Richard A Steenbergen]

On Mon, Jan 09, 2006 at 12:42:51PM -0800, Gregory Maxwell wrote:
> > -----Original Message-----
> > A good description of the process is found at 
> > http://juniper.cluepon.net/index.php/J-cell
> > 
> > But I disagree with the wiki text on
> > http://juniper.cluepon.net/index.php/FPC_Bandwidth that 
> > states 3.2 Gbps as a realistic worst case, because not all 
> > packets will have sizes that are multiple of 64 bytes, which 
> > is the only case that would achieve the 3.2 Gbps maximum.
> > This reference 
> > (http://www.nlanr.net/NA/Learn/Gm/pktsizes.html) states that 
> > 40 byte packets account for 40% of the packets flowing 
> > around; 40 byte packets require 16 byte paddings for each 
> > J-Cell, so if all other packets would be 64 byte multiples, 
> > performance would already been limited to 2.72 Gbps.
> > 
> > I agree that 2.5 Gbps is a safe bet here.
> 
> What PIC configuration will physically allow more than 6.1MPPSish per
> FPC to physically enter the router? Think carefully and consider all
> overhead and you will be enlightened.

Trying to figure out a realistic worst-case bandwidth capacity based on a 
packet/sec capacity is silly at best, but if we really are going to do 
that, lets at least acknowledge that 2.5Gbps isn't it.

The raw bandwidth limitation is 4Gbps, so the 3.2Gbps number is already 
calculated to be exclusive of J-cell HEADER overhead but not of J-cell 
PADDING overhead. This means that the actual bandwidth is 3.2Gbps when 
evenly divided into 64 byte chunks, which obviously means that the worst 
case packet is going to be 65 bytes. 65 bytes of IP consumes 128 bytes of 
J-cell bandwidth, which means that our 3.2Gbps is consumed in just 
1.625Gbps of IP.

I think the 2.5Gbps "safe bet" stuff comes from the assumption that 
Juniper made the FPC "OC48 safe" but not "4xGE safe", which is not true. 
SONET has less overhead with small packets, so an OC48 with 40 byte 
packets will do 6.1Mpps, while 4xGE with 46 byte packets will top out at 
4*1.448Mpps = 5.792Mpps. OC48 w/65 byte packets will do 4.5Mpps, which is 
of course more than the 3.125Mpps of capacity you'll get off an FPC1 
burning 2 cells per packet.

Also, in reference to the URL above, that packet size distribution data 
appears to be over 10 years old. The only part that is probably still 
relevent on today's internet is that half of the packets are ACK's, which 
means you should never try to base your utilization calculations on 
averages or "internet mix" numbers. What you're really seeing is the vast 
majority of your "content" packets at 1500 bytes in one direction, being 
acknowledged by 40 byte "ack" packets in the other direction, and a bunch 
of misc sizes for all that "other" stuff (gaming, http requests, etc). Of 
course the numbers 40 and 1500 are approximate, since even Windows now 
tries to negotiate at least SACK, and sometimes RFC1323 timestamps and 
window scaling too. If you are a web hoster, your 1500's will all be piled 
up in one direction and your 40's will all be piled up in the other 
direction, so be sure to make your inbound packet/sec calculations 
assuming approx 40 byte packets. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)