[c-nsp] ASA5520 latency & OSPF drops

[Adam Greene]

Hi,

 

We are having a problem with high latency and OSPF drops on an ASA5520. 

 

The portion of our network in question is connected as follows: 

 

Internal Network---3750---2950G---ASA5520---2950G---2921---External World

 

The two 2950G's shown above are actually the same device; we are using VLANs
to segment the traffic. 

 

We're running OSPF between the 3750 and the ASA5520, and between the ASA5520
and the 2921. 

 

Every so often (it started three months ago, about once per month, now it's
about once per week, but it's not regular), we're getting very high latency
on pings from our Internal Network to the ASA5520, and the OSPF adjacency
between the 3750 and the ASA5520 is dropping. The issue was lasting about 60
seconds each time up to this morning, when it lasted about 3 hours. Ugh!

 

Pings from the Internal Network to the 3750 and 2950G are fine. 

 

The OSPF adjacency between the ASA5520 and the 2921 is not affected.

 

This would seem to suggest an issue between the 2950G and the ASA5520.

 

There are some input errors showing on the inside interface of the ASA5520,
but very few compared with the traffic that passes through the interface
(0.009%). There is no evidence of errors on the 2950G interface(s), even
when "show controllers Ethernet-controller" is issued.  

 

The 3750 is showing:

 

Feb  1 06:12:03: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
from LOADING to FULL, Loading Done

Feb  1 06:17:03: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
from LOADING to FULL, Loading Done

Feb  1 06:18:54: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
from LOADING to FULL, Loading Done

Feb  1 07:40:35: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
from LOADING to FULL, Loading Done

Feb  1 07:46:55: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
from LOADING to FULL, Loading Done

Feb  1 07:59:46: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
from LOADING to FULL, Loading Done

 

Strangely, it is not showing any FULL to DOWN events. 

 

The ASA is not logging OSPF drops, but "show ospf neighbor" does show that
the neighbor has only been up since the last drop. 

 

We do not see any evidence of CPU or traffic spikes (either in terms of
bandwidth, connection counts, or number of unicast packets traversing the
link). RAM on the ASA5520 went up very slightly during this morning's
events, but hardly enough to care about.

 

MTU is set to 1500 on all implicated 3750, 2950G and ASA interfaces.

 

We are rather stumped. The ASA is running 8.2(4) . we're thinking of
upgrading to 8.2(5). We are also considering:

-          bypass the 2950G 

-          replace the ASA5520 with a spare

-          replace the 3750 with a spare

 

All these options imply 3am maintenance windows. 

 

Any ideas before we start to have a few sleepless nights? :)

 

Thanks,

Adam