[c-nsp] 7507 and 5500 arp oddities

[James Saker]

I have a "works in the lab but not in the field" issue with a 7507
etherchannel'ed out to a Catalyst 5500 with VLANs to each 5500 port for
router port control. Under production traffic load (2-5 Mbps), the 7507
begins losing arp representations of the remote systems. Arp associations
even with direct-connected systems take 5+ minutes. 

As the same topology works fine with a Cisco 2651 connected to a (gasp)
Catalyst 2800 switch, I'm wondering if I've either got an arp issue or if my
ugly "eight /30 links on a single VLAN, single port" legacy design is
confusing the 7507/5500. Or perhaps there's some 7507/5500 tuning magic I've
overlooked somewhere.

A high level of the design is:

Upstream 7507-->MPLS Core

MPLS Core-(dual OSPF Ethernet links)-->7507

7507--(dual 100BASET Etherchannel, VLAN trunk)-->5500

5500-(unique VLAN per port, e.g. VLAN 707 for interface 7/07)-> remotes

On one local VLAN, I have a /29 network for connecting tech laptops. Today
it took more than 5 minutes to have ARP associate me with the 7507. I could
immediately ping laptop to laptop on the LAN, but the 5500 and 7507 wouldn't
let me past. ARP table on 7507 actually had an entry for me though, but no
ping/reply and no routing past 7507.

Interestingly, the 5500 has suddenly become lost to the 7507 in the
management VLAN talking to sc0 - worked fine in lab. No changes in config,
but now it refuses to talk. There are no other VLANs connected to this
switch (VLAN only used to fan out from 7500), so it's not as if I'm
encountering another VLAN out there competing for control.

Another intrigue is that the 7507 lost some but not all of my remotes on one
VLAN/port, VLAN 709 (catalyst 7/09) had eight /30 links via a
point-multipoint radio system; e.g. 10.1.1.0/30 goes to one remote 2650
router over a multipoint radio system. The 7507/5500 is connected to an
access point here, and each remote radio talks to a 2650 router. Three of
these routers got lost overnight, while the rest of the /30 subnets were
still visible to the 7507. Getting into the radios, all was happy - for some
reason, the 7507/5500 seems to be losing these connected routes. Naturally,
in the lab it all worked fine for two months and I never saw this behavior.
Any thoughts are greatly appreciated!

Jamie