[c-nsp] C7301 dropping OSPF
McDonald Richards
mcdonald.richards at gmail.com
Sat Apr 25 05:13:49 EDT 2009
At the risk of over simplifying the issue - I'd suggest you increase your
input queue on both devices to at least 1000. Looks like the 75 packet queue
is not enough for the pps load on the device and other process switched
packets may be causing your OSPF messages to be tail dropped during periods
of high load causing your adjacencies to reset.
Increase the input queue on both devices, clear counters and monitor for
input queue drops. The overruns on the interface may also indicate the
hardware was under serious load at one point. Again - reset the counters and
monitor so you have a reference point so you can try and determine the root
cause.
On Sat, Apr 25, 2009 at 4:17 AM, Bill Blackford <BBlackford at nwresd.k12.or.us
> wrote:
> wsc-rtr-7301#sh int gi0/0
> GigabitEthernet0/0 is up, line protocol is up
> Hardware is BCM1250 Internal MAC, address is ####.####.#### (bia
> ####.####.####)
> Description: Internet
> Internet address is x.x.x.x/30
> MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
> reliability 255/255, txload 4/255, rxload 30/255
> Encapsulation ARPA, loopback not set
> Keepalive set (10 sec)
> Full-duplex, 1000Mb/s, media type is RJ45
> output flow-control is unsupported, input flow-control is unsupported
> ARP type: ARPA, ARP Timeout 04:00:00
> Last input 00:00:00, output 00:00:00, output hang never
> Last clearing of "show interface" counters never
> Input queue: 0/75/116200/85873 (size/max/drops/flushes); Total output
> drops: 1
> Queueing strategy: fifo
> Output queue: 0/40 (size/max)
> 5 minute input rate 120394000 bits/sec, 13851 packets/sec
> 5 minute output rate 17119000 bits/sec, 9050 packets/sec
> 926342756 packets input, 1996679340 bytes, 0 no buffer
> Received 1 broadcasts, 0 runts, 0 giants, 33 throttles
> 0 input errors, 0 CRC, 0 frame, 241700 overrun, 0 ignored
> 0 watchdog, 812170 multicast, 0 pause input
> 0 input packets with dribble condition detected
> 3157683501 packets output, 4172533168 bytes, 0 underruns
> 4 output errors, 0 collisions, 3 interface resets
> 0 babbles, 0 late collision, 0 deferred
> 4 lost carrier, 0 no carrier, 0 pause output
> 0 output buffer failures, 0 output buffers swapped out
>
> wsc-rtr-7301#sh int gi0/1
> GigabitEthernet0/1 is up, line protocol is up
> Hardware is BCM1250 Internal MAC, address is ####.####.#### (bia
> ####.####.####)
> Description: NWRESD WAN & all SDs
> Internet address is x.x.x.x/24
> MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
> reliability 255/255, txload 27/255, rxload 3/255
> Encapsulation ARPA, loopback not set
> Keepalive set (10 sec)
> Full-duplex, 1000Mb/s, media type is RJ45
> output flow-control is unsupported, input flow-control is unsupported
> ARP type: ARPA, ARP Timeout 04:00:00
> Last input 00:00:00, output 00:00:00, output hang never
> Last clearing of "show interface" counters never
> Input queue: 1/75/168340/202 (size/max/drops/flushes); Total output drops:
> 5
> Queueing strategy: fifo
> Output queue: 0/40 (size/max)
> 5 minute input rate 14421000 bits/sec, 8824 packets/sec
> 5 minute output rate 109796000 bits/sec, 13114 packets/sec
> 3350123857 packets input, 3632867472 bytes, 137 no buffer
> Received 285023 broadcasts, 0 runts, 0 giants, 2375 throttles
> 0 input errors, 0 CRC, 0 frame, 164839 overrun, 0 ignored
> 0 watchdog, 697983 multicast, 0 pause input
> 0 input packets with dribble condition detected
> 1133783666 packets output, 2917422301 bytes, 0 underruns
> 4 output errors, 0 collisions, 3 interface resets
> 0 babbles, 0 late collision, 0 deferred
> 4 lost carrier, 0 no carrier, 0 pause output
> 0 output buffer failures, 0 output buffers swapped out
>
>
>
> wsc-rtr-7301#sh ip ospf interface
> Loopback10 is up, line protocol is up
> Internet Address x.x.x.x/32, Area 0
> Process ID 5794, Router ID x.x.x.x, Network Type LOOPBACK, Cost: 1
> Loopback interface is treated as a stub Host
> GigabitEthernet0/2 is up, line protocol is up
> n/a
> GigabitEthernet0/0 is up, line protocol is up
> Internet Address x.x.x.x/30, Area 0
> Process ID 5794, Router ID x.x.x.x, Network Type BROADCAST, Cost: 1
> Transmit Delay is 1 sec, State BDR, Priority 25
> Designated Router (ID) x.x.x.x, Interface address x.x.x.x
> Backup Designated router (ID) x.x.x.x, Interface address x.x.x.x
> Timer intervals configured, Hello 2, Dead 8, Wait 8, Retransmit 1
> oob-resync timeout 40
> Hello due in 00:00:01
> Supports Link-local Signaling (LLS)
> Index 2/2, flood queue length 0
> Next 0x0(0)/0x0(0)
> Last flood scan length is 1, maximum is 32
> Last flood scan time is 0 msec, maximum is 52 msec
> Neighbor Count is 1, Adjacent neighbor count is 1
> Adjacent with neighbor x.x.x.x (Designated Router)
> Suppress hello for 0 neighbor(s)
> Message digest authentication enabled
> Youngest key id is 5
> GigabitEthernet0/1 is up, line protocol is up
> Internet Address x.x.x.x/24, Area 0
> Process ID 5794, Router ID x.x.x.x, Network Type BROADCAST, Cost: 1
> Transmit Delay is 1 sec, State DR, Priority 30
> Designated Router (ID) x.x.x.x, Interface address x.x.x.x
> Backup Designated router (ID) x.x.x.x, Interface address x.x.x.x
> Timer intervals configured, Hello 2, Dead 8, Wait 8, Retransmit 1
> oob-resync timeout 40
> Hello due in 00:00:00
> Supports Link-local Signaling (LLS)
> Index 1/1, flood queue length 0
> Next 0x0(0)/0x0(0)
> Last flood scan length is 3, maximum is 4
> Last flood scan time is 0 msec, maximum is 4 msec
> Neighbor Count is 1, Adjacent neighbor count is 1
> Adjacent with neighbor x.x.x.x (Backup Designated Router)
> Suppress hello for 0 neighbor(s)
> Message digest authentication enabled
> Youngest key id is 5
>
>
>
>
> -----Original Message-----
> From: Mike Louis [mailto:MLouis at nwnit.com]
> Sent: Friday, April 24, 2009 10:23 AM
> To: Bill Blackford; cisco-nsp at puck.nether.net
> Subject: RE: [c-nsp] C7301 dropping OSPF
>
> Bill
>
> Do you have multicast routing enabled on your network? If so what multicast
> addresses are you using?
>
> Can you do a show interface on the routed interfaces and post? Show ip ospf
> interface as well if you would.
>
> -----Original Message-----
> From: Bill Blackford <BBlackford at nwresd.k12.or.us>
> Sent: Friday, April 24, 2009 1:17 PM
> To: cisco-nsp at puck.nether.net <cisco-nsp at puck.nether.net>
> Subject: [c-nsp] C7301 dropping OSPF
>
>
> PROBLEM:
> 7301 dropping OSPF adjacencies. The log is showing the following messages:
>
> Apr 23 12:20:01 c7301 392: 000398: Apr 23 12:20:00.016 PDT: %OSPF-5-ADJCHG:
> Process 5794, Nbr x.x.x.x on GigabitEthernet0/0 from LOADING to FULL,
> Loading Done
> Apr 23 12:20:01 c7301 393: 000399: Apr 23 12:20:00.016 PDT: %OSPF-5-ADJCHG:
> Process 5794, Nbr x.x.x.x on GigabitEthernet0/1 from LOADING to FULL,
> Loading Done
>
>
> During this, all traffic moves over to my other router. BGP peers stay up,
> but IGP routes get dumped.
> The router sees about 200M and 30k PPS on each interface at peak times.
> CPU gets to about 55% at peak. This issue is occurring near, but not always
> at peak.
> Code is older, disk0:c7301-js-mz.123-14.T3.bin. But this issue just started
> showing up. It has me a bit concerned as we just had a series of power
> events with a misbehaved UPS.
>
>
> I can ping the multicast address fine:
> wsc-rtr-7301#ping 224.0.0.5
>
> Type escape sequence to abort.
> Sending 1, 100-byte ICMP Echos to 224.0.0.5, timeout is 2 seconds:
>
> Reply to request 0 from lo.bdr1.fqdn (x.x.x.x), 1 ms Reply to request 0
> from ge-1-0-6.agr1.fqdn (x.x.x.x), 4 ms Reply to request 0 from
> ge-0-2-v100.bdr1.fqdn (x.x.x.x), 1 ms
>
> wsc-rtr-7301#sh ip os n
>
> Neighbor ID Pri State Dead Time Address Interface
> x.x.x.x 255 FULL/DR 00:00:06 x.x.x.x GigabitEthernet0/0
> y.y.y.y 25 FULL/BDR 00:00:07 x.x.x.x
> GigabitEthernet0/1
>
> The interfaces includes the following:
>
> int gi0/0
> ip ospf hello-interval 2
> ip ospf priority 25
> ip ospf retransmit-interval 1
>
>
>
> int gi0/1
> ip ospf hello-interval 2
> ip ospf priority 30
> ip ospf retransmit-interval 1
>
>
> Thank you in advance for any help.
>
> -b
>
> --
> Bill Blackford
> Senior Network Engineer
> NWRESD
>
> my /home away from home
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
> Note: This message and any attachments is intended solely for the use of
> the individual or entity to which it is addressed and may contain
> information that is non-public, proprietary, legally privileged,
> confidential, and/or exempt from disclosure. If you are not the intended
> recipient, you are hereby notified that any use, dissemination,
> distribution, or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the original
> sender immediately by telephone or return email and destroy or delete this
> message along with any attachments immediately.
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
More information about the cisco-nsp
mailing list