[c-nsp] ASA5520 latency & OSPF drops

[David White, Jr. (dwhitejr)]

Hi Adam,

So, the symptoms are high latency from internal network to Inside of
ASA's interface?
And during this problem, the switch appears to be re-establishing the
OSPF neighbor?

It wasn't clear to me if you were also seeing packet loss or not.

A suggestion to narrow down some things:
If the 2950G has an L3 interface in the same segment as the ASA's inside
interface, then do pings from the ASA (and likewise from the switch)
show the latency?

If there is no L3 interface on the 2950, then perform this test from a
device local to the network.

One thing you want to do is determine if OSPF is the cause, or just a
victim of a different problem.  Checking from local devices (or adding
static routes) will help eliminate OSPF.

For some local issues, things that come to mind:
 * Duplicate IP addresses
 * L2 loops
 * Spanning-tree changes

Also, on the ASA, look at the output of "show asp drop".  You might want
to clear it, and then look at this output after the next occurrence of
the issue.

Sincerely,

David.

On 2/1/2014 11:27 AM, Adam Greene wrote:
> Hi,
>
>  
>
> We are having a problem with high latency and OSPF drops on an ASA5520. 
>
>  
>
> The portion of our network in question is connected as follows: 
>
>  
>
> Internal Network---3750---2950G---ASA5520---2950G---2921---External World
>
>  
>
> The two 2950G's shown above are actually the same device; we are using VLANs
> to segment the traffic. 
>
>  
>
> We're running OSPF between the 3750 and the ASA5520, and between the ASA5520
> and the 2921. 
>
>  
>
> Every so often (it started three months ago, about once per month, now it's
> about once per week, but it's not regular), we're getting very high latency
> on pings from our Internal Network to the ASA5520, and the OSPF adjacency
> between the 3750 and the ASA5520 is dropping. The issue was lasting about 60
> seconds each time up to this morning, when it lasted about 3 hours. Ugh!
>
>  
>
> Pings from the Internal Network to the 3750 and 2950G are fine. 
>
>  
>
> The OSPF adjacency between the ASA5520 and the 2921 is not affected.
>
>  
>
> This would seem to suggest an issue between the 2950G and the ASA5520.
>
>  
>
> There are some input errors showing on the inside interface of the ASA5520,
> but very few compared with the traffic that passes through the interface
> (0.009%). There is no evidence of errors on the 2950G interface(s), even
> when "show controllers Ethernet-controller" is issued.  
>
>  
>
> The 3750 is showing:
>
>  
>
> Feb  1 06:12:03: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
> from LOADING to FULL, Loading Done
>
> Feb  1 06:17:03: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
> from LOADING to FULL, Loading Done
>
> Feb  1 06:18:54: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
> from LOADING to FULL, Loading Done
>
> Feb  1 07:40:35: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
> from LOADING to FULL, Loading Done
>
> Feb  1 07:46:55: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
> from LOADING to FULL, Loading Done
>
> Feb  1 07:59:46: %OSPF-5-ADJCHG: Process 2, Nbr x.x.x.x on FastEthernet1/0/1
> from LOADING to FULL, Loading Done
>
>  
>
> Strangely, it is not showing any FULL to DOWN events. 
>
>  
>
> The ASA is not logging OSPF drops, but "show ospf neighbor" does show that
> the neighbor has only been up since the last drop. 
>
>  
>
> We do not see any evidence of CPU or traffic spikes (either in terms of
> bandwidth, connection counts, or number of unicast packets traversing the
> link). RAM on the ASA5520 went up very slightly during this morning's
> events, but hardly enough to care about.
>
>  
>
> MTU is set to 1500 on all implicated 3750, 2950G and ASA interfaces.
>
>  
>
> We are rather stumped. The ASA is running 8.2(4) . we're thinking of
> upgrading to 8.2(5). We are also considering:
>
> -          bypass the 2950G 
>
> -          replace the ASA5520 with a spare
>
> -          replace the 3750 with a spare
>
>  
>
> All these options imply 3am maintenance windows. 
>
>  
>
> Any ideas before we start to have a few sleepless nights? :)
>
>  
>
> Thanks,
>
> Adam
>
>  
>
>  
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/