[c-nsp] TCP behavior under strict CAR rate-limiting

[bill fumerola]

On Thu, Jun 19, 2008 at 04:16:19PM -0700, Christopher Hunt wrote:
>    It would appear from the sender's counters and from the snmp checks 
> on the router interface that the interface never hits 10mbps even for a 
> second, but the rate-limiting counters do show tail drops. I guess it is 
> difficult to get the sub-second granularity necessary to see the process 
> in action and by the time the counters are hit again, they've averaged 
> out over the second.  

instantaneous counters have this problem. clearing before each experiment
and using relative wall clock time and absolute counters are an option.

>                       I know, for example that the the stats provided by 
> "ifconfig" under RedHat only seem to update every 1-2 seconds.  

often this is because they're only polled from the ethernet hardware's
registers periodically. other counters (like IP protocol statistics)
exist in kernel memory.

> Similarly SNMP is polled at most each second and while it shows no 
> spikes, the interface must be receiving spikes > CAR (even if only for 
> microseconds?)in order to drop packets, right?  

if the backing data for the SNMP counters is hardware depdendent, the
counters will reflect that. otherwise, things are implementation dependent.

>                                                 I wonder how the 
> rate-limiting counters really work with the Cisco.  It obviously doesn't 
> do the math each second, but instead i guess it does the math each time 
> a packet arrives and the 1 second inteface counters  obviously don't 
> show burst that last a few microseconds.

i don't know how the internals work. there are a number of different
ways to implement this. even across multiple platforms there will be
differences due to HZ, hardware v. software classification, etc

i'm sure you've already read it, but:
http://www.cisco.com/en/US/docs/ios/12_2/qos/configuration/guide/fqos_c.html

has quite a bit of detail. outside of source code, the implementation
specifics are magical. which brings us to:
>    I wish i could see the cwnd in action, but I guess I'll have to 
> content myself with what we can see.  Bill et al., thanks very much for 
> checking this out.  I hope to be useful to others some day ;-)

if you're really interested, using freebsd you can compile a kernel with
'options TCPDEBUG' to export this information. see also:

trpt(8)                  - transliterate protocol trace

finally, the folks on the IETF tcpm list are helpful, but they also eat
sleep and breathe tcp state, timers, congestion, etc. if you have specific
questions about specific conditions, they can clarify. i'd read the
archives before posting - a lot of these things have been covered before.

-- bill