[c-nsp] 3550 12.2.25SEB4 ->12.2.44SE breaks OSPF?

[randal k]

NSP'ers,
We recently did a maintenance to upgrade some of our aging 3550s to
newer code. After the IOS upgrade, the switch came back online and
formed OSPF adjacencies & exchanged traffic with all of our other
switches, but could not maintain one with any of our routers.

For example, the upgraded IOS switch was able to get some OSPF action
with a 3750, 4 other 3550s of various code versions, and a 3560g with
no problem. However, it would not hold one open with a 7206/NPE-G1, a
6509/Sup7203bxl and a 3845.

Gist is that it would open, the adjacency would form, and then after
25 or so retransmits the problem 3550 would close the session. The
peering routers all thought things were OK. Here is a short snippet of
the logs:

Dec  5 00:12:40 C3550 214: 00:08:11: OSPF: Rcv LS UPD from CISCO6509
on Vlan101 length 92 LSA count 1
Dec  5 00:12:40 lo0.c6509.fac01.cos 1007: Dec  5 00:12:39 MST:
%OSPF-5-ADJCHG: Process 10, Nbr C3550 on Vlan101 from LOADING to FULL,
Loading Done
Dec  5 00:12:45 C3550 215: 00:08:15: OSPF: Retransmitting request to
CISCO3845 on Vlan101
Dec  5 00:12:45 C3550 216: 00:08:15: OSPF: Send LS REQ to CISCO3845
length 1476 LSA count 123
Dec  5 00:12:45 C3550 217: 00:08:15: OSPF: Retransmitting request to
CISCO6509 on Vlan101
Dec  5 00:12:45 C3550 218: 00:08:15: OSPF: Send LS REQ to CISCO6509
length 1476 LSA count 123
*repeat for 2 minutes*
Dec  5 00:14:30 c3550 303: 00:10:00: OSPF: Send LS REQ to CISCO6509
length 1476 LSA count 123
Dec  5 00:14:35 c3550 304: 00:10:05: OSPF: Retransmitting request to
CISCO3845 on Vlan101
Dec  5 00:14:35 c3550 305: 00:10:05: OSPF: Send LS REQ to CISCO3845
length 1476 LSA count 123
Dec  5 00:14:35 c3550 306: 00:10:05: OSPF: Retransmitting request to
CISCO6509 on Vlan101
Dec  5 00:14:35 c3550 307: 00:10:05: OSPF: Send LS REQ to CISCO6509
length 1476 LSA count 123
Dec  5 00:14:39 c3550 308: 00:10:10: OSPF: Killing nbr CISCO3845 on
Vlan101 due to excessive (25) retransmissions

The 3550 states that it goes into loading/dr, loading/bdr & exstart
for the 6509,3845 & 7206 respectively. Our config is *extremely*
simple in this situation - loopback+3-4 routes per box, active
interfaces everywhere, handful of VLANs, nothing complex at all.

It is not an MTU issue; all MTUs have been verified and debugging it
shows it accepting the MTUs. (It's a legacy setup, I presume MTU=1530
is for QinQ/MPLS)
Dec  5 00:33:45 C35502655: 00:29:15: OSPF: Rcv DBD from CISCO3845 on
Vlan101 seq 0xB9C opt 0x52 flag 0x2 len 572  mtu 1500 state EXCHANGE
Dec  5 00:33:45 C35502657: 00:29:15: OSPF: Rcv DBD from CISCO3845 on
Vlan101 seq 0xB9D opt 0x52 flag 0x0 len 32  mtu 1500 state EXCHANGE
Dec  5 00:33:52 C35502675: 00:29:23: OSPF: Rcv DBD from CISCO6509 on
Vlan101 seq 0xAC9 opt 0x52 flag 0x7 len 32  mtu 1530 state EXSTART
Dec  5 00:33:52 C35502677: 00:29:23: OSPF: Rcv DBD from CISCO6509 on
Vlan101 seq 0x1956 opt 0x52 flag 0x2 len 1452  mtu 1530 state EXSTART
Dec  5 00:33:52 C35502680: 00:29:23: OSPF: Rcv DBD from CISCO6509 on
Vlan101 seq 0x1957 opt 0x52 flag 0x2 len 1452  mtu 1530 state EXCHANGE


Rolling back from 12.2.44SE -> 12.2.25SEB4 resolved the issue with no
config changes.

Any ideas? I'm more than happy to send over unedited logfiles offline.

Thanks,
Randal