[j-nsp] mysterious microbursts ?

[Alexandre Snarskii]

Hi!

During routine maintenance on my ex-3200 switches I found that 
some devices, connected with 100mbit/full-duplex observe minor
packet loss. Well, there is nothing too unexpected, you can 
see it mostly every time when traffic enters switch using 1ge (2x1ge
aggregate in my case) link and exits using 100mbit, switch ports
just do not have enough buffer space to handle bursts well, but... 
That's where the strangeness begins: 
a) these 100mbit ports used to connect VoIP devices handling RTP 
traffic, so there should be no bursts at all.
b) there are more than 30 such devices in my network, and I see 
that loss only on half of them - on that half that is connected 
to even-numbered switches, there are no packet loss on devices 
connected to odd-numbered ones.

Logical topology is just "a vlan connected to a router (two routers
in VRRP master/slave scenario)", physical is classic ring (some rings 
actually, each odd-numbered access switch connected to coresw1, 
even-numbered - to coresw2): 

  router1                  router2(backup, no traffic right now)
    |                        |
  coresw1 ================ coresw2
    |                        |
    |                        |
  accsw1 -------- x ------ accsw2

link between core switches is 10ge, links core-access and access-access
is 2x ge aggregated ethernet, access-access links are blocked by STP. 
Switches is ex3200-24t(core) and ex3200-48t(access), running 9.3R4.4.

Well, these losses is about 0.1% and I can live with that, but
I'd like to have any idea on why it happens only on half of 
devices, and I don't have one :( 
Any clue ? 

Example interface with errors: 

Physical interface: ge-0/0/26, Enabled, Physical link is Up
  Interface index: 165, SNMP ifIndex: 174, Generation: 168
  Link-level type: Ethernet, MTU: 1514, Speed: 100mbps, Duplex: Full-Duplex, MAC-REWRITE Error: None,
  Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Disabled,
  Remote fault: Online
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x0
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 00:19:e2:53:66:9a, Hardware address: 00:19:e2:53:66:9a
  Last flapped   : 2010-03-05 04:36:34 PST (1w1d 22:21 ago)
  Statistics last cleared: 2010-03-11 06:44:54 PST (2d 20:13 ago)
  Traffic statistics:
   Input  bytes  :          88312256790                    0 bps
   Output bytes  :          88616684644                 5984 bps
   Input  packets:            443797702                    0 pps
   Output packets:            412072442                   10 pps
   IPv6 transit statistics:
    Input  bytes  :                   0 
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0,
    L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0,
    MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0 best-effort                    0            411934501               454642
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    1 assured-forw                   0                    0                    0
    5 expedited-fo                   0                    0                    0
    7 network-cont                   0               137336                    0
  Active alarms  : None
  Active defects : None
  MAC statistics:                      Receive         Transmit
    Total octets                   88312256790      88616684644
    Total packets                    443797702        412072442
    Unicast packets                  443796941        410096125
    Broadcast packets                      761           447035
    Multicast packets                        0          1529282
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    Code violations                          0
  Autonegotiation information:
    Negotiation status: Incomplete
  Packet Forwarding Engine configuration:
    Destination slot: 0
    Direction : Output 
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95       95000000    95             NA      low    none
    7 network-control         5        5000000     5             NA      low    none