[c-nsp] Link Problem - 6509 to GSR
Paul Stewart
pstewart at nexicomgroup.net
Mon Oct 23 12:33:25 EDT 2006
core1-gsr-mb#sh controller psar tofab
To Fabric Stats:
---------------
Slot Tx-pkts Tx-TH-pkts Tx-dropped Tx-DMA
0 0 786061 0 786061
1 0 2280956 0 2280956
3 2938767 1304541 0 4243259
4 882883 1298236 0 2181119
5 1234 429347 0 430581
6 11 912464 9566 910235
7 8755234 2238749 0 10993983
8 0 826558 0 826558
mcast 0 11368 0 11368
To Fabric Errors:
----------------
Failed sends because of no header bufs - 0
Failed sends because of no payload bufs - 0
SRAM ECC single-bit (corrected) errors - 0
DMA errors - 0
Fusilli Parity errors - 0
core1-gsr-mb#sh controller psar tofab
To Fabric Stats:
---------------
Slot Tx-pkts Tx-TH-pkts Tx-dropped Tx-DMA
0 0 786061 0 786061
1 0 2280956 0 2280956
3 2938767 1304541 0 4243259
4 882883 1298236 0 2181119
5 1234 429347 0 430581
6 11 912464 9566 910235
7 8755234 2238749 0 10993983
8 0 826558 0 826558
mcast 0 11368 0 11368
To Fabric Errors:
----------------
Failed sends because of no header bufs - 0
Failed sends because of no payload bufs - 0
SRAM ECC single-bit (corrected) errors - 0
DMA errors - 0
Fusilli Parity errors - 0
core1-gsr-mb#sh controller psar drop
To/From Fabric dropped packets:
------------------------------
Slot Tx-pkt Rx-pkt
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 9566 0
7 0 0
8 0 0
9 0 0
10 0 0
11 0 0
12 0 0
13 0 0
14 0 0
15 0 0
mcast 0 3
-----Original Message-----
From: Robbie.Jacka at amsouth.com [mailto:Robbie.Jacka at amsouth.com]
Sent: Monday, October 23, 2006 12:18 PM
To: Paul Stewart
Cc: cisco-nsp at puck.nether.net; cisco-nsp-bounces at puck.nether.net;
Robbie.Jacka at amsouth.com
Subject: Re: [c-nsp] Link Problem - 6509 to GSR
Do a 'sh controller psar tofab' and 'sh controller psar drop' - sounds
like slot 6 is maybe blowing up your backend queuing.
--
Robbie Jacka / robbie.jacka at amsouth.com
AmSouth Bank / Information Security Operations
ofc:205.261.6486 / fax:205.261.6424 / ROC 3N5-12
"Paul Stewart"
<pstewart at nexicom
group.net>
To
Sent by: Robbie.Jacka at amsouth.com
cisco-nsp-bounces
cc
@puck.nether.net
cisco-nsp-bounces at puck.nether.net,
cisco-nsp at puck.nether.net
Subject
10/23/2006 11:08 Re: [c-nsp] Link Problem - 6509
to
AM GSR
Just an update on this issue...
It looks like hardware is one problem... have replaced one of the GigE
blades on the GSR side as was getting the following:
%FIB-4-FIBXDRINV: Invalid format. invalid if_number
Oct 23 11:34:26: %PRP-3-CHP_DESCQ_FULL: Chopper desc queue 6 full - enq
1733452
deq 1732430 blog 1022
-Traceback= 202820 202928 52C528 52CF00 52D08C 34D10C 34D1C0 5EDC24
601E08 5F18B
C 5F2334
core1-gsr-mb#
Oct 23 11:34:31: %PRP-3-CHP_DESCQ_FULL: Chopper desc queue 6 full - enq
1733452
deq 1732430 blog 1022
-Traceback= 202820 202928 52C528 52CF00 52D08C 34D10C 34D1C0 5EDC24
601E08 5F18B
C 5F2334 2EA52C
Oct 23 11:34:36: %PRP-3-CHP_DESCQ_FULL: Chopper desc queue 6 full - enq
1733452
deq 1732430 blog 1022
-Traceback= 202820 202928 52C528 52CF00 52D08C 34D10C 34D1C0 5EDC24
601E08 5F18B
C 5F2334 2EA52C
Oct 23 11:34:41: %PRP-3-CHP_DESCQ_FULL: Chopper desc queue 6 full - enq
1733452
deq 1732430 blog 1022
-Traceback= 202820 202928 52C528 52CF00 52D08C 34D10C 34D1C0 5EDC24
601E08 5F18B
C 5F2334 2EA52C
Oct 23 11:34:46: %COSGRP-3-UNEXP: retry queue flush -Traceback= 202820
202928 361D30 369470 36CC74 2EA52C Oct 23 11:34:46: %LINK-5-CHANGED:
Interface GigabitEthernet6/0, changed state to administratively down
Oct 23 11:34:46: %LINEPROTO-5-UPDOWN: Line protocol on Interface
GigabitEthernet 6/0, changed state to down
I can't find a lot of information on this at cisco.com - anyone care to
take a stab at it? The key error seems to be around this:
core1-gsr-mb#
SEC 1:.Oct 23 12:01:44: %FIB-4-FIBXDRINV: Invalid format. invalid
if_number SLOT 6:00:00:13: %SYS-5-RESTART: System restarted -- Cisco
Internetwork Operating System Software IOS (tm) GS Software (GLC1-LC-M),
Version 12.0(32)S4, RELEASE SOFTWARE
(fc1)
Technical Support: http://www.cisco.com/techsupport Copyright (c)
1986-2006 by cisco Systems, Inc.
Compiled Wed 16-Aug-06 18:04 by leccese
As you can see, the linecard reloads itself after crapping out... is
this a memory issue?
The GSR has dual PRP-2 processors if that helps...
The original issue still exists even with a new linecard installed...
Thanks,
Paul
-----Original Message-----
From: Robbie.Jacka at amsouth.com [mailto:Robbie.Jacka at amsouth.com]
Sent: Monday, October 23, 2006 10:50 AM
To: Paul Stewart
Cc: cisco-nsp at puck.nether.net; cisco-nsp-bounces at puck.nether.net;
Robbie.Jacka at amsouth.com
Subject: Re: [c-nsp] Link Problem - 6509 to GSR
What's your MAC timeout configured as? Do you have UDLD configured on
either side? Why is flow control configured on the 6509? Also, notice on
Gi9/6 on the 6509:
Last input 2d23h, output 00:00:33, output hang never
I glossed over that when I was looking - guess I looked at Vl30 and not
the phy layer.
--
Robbie Jacka / robbie.jacka at amsouth.com
AmSouth Bank / Information Security Operations
ofc:205.261.6486 / fax:205.261.6424 / ROC 3N5-12
"Paul Stewart"
<pstewart at nexicom
group.net>
To
Sent by: Robbie.Jacka at amsouth.com
cisco-nsp-bounces
cc
@puck.nether.net
cisco-nsp-bounces at puck.nether.net,
cisco-nsp at puck.nether.net
Subject
10/23/2006 09:42 Re: [c-nsp] Link Problem - 6509
to
AM GSR
Thanks for the reply...
I get a MAC on the 6509:
core1-rtr-mb#sh mac-address-table Vlan 30
Legend: * - primary entry
vlan mac address type learn ports
------+----------------+--------+-----+--------------------------
* 30 0050.e28a.bda9 dynamic Yes Gi9/6
* 30 0004.defd.f40a static No Router
On the GSR:
core1-gsr-mb#sh arp | inc 5/0
Internet 10.250.250.2 - 0050.e28a.bda9 ARPA
GigabitEthernet5/0
Internet 10.250.250.1 0 0004.defd.f40a ARPA
GigabitEthernet5/0
-----Original Message-----
From: Robbie.Jacka at amsouth.com [mailto:Robbie.Jacka at amsouth.com]
Sent: Monday, October 23, 2006 10:32 AM
To: Paul Stewart
Cc: cisco-nsp at puck.nether.net; cisco-nsp-bounces at puck.nether.net
Subject: Re: [c-nsp] Link Problem - 6509 to GSR
Packets appear to be traversing the link - have you checked the MAC
table on the 6509, or ARP table on the GSR? Checked CDP neighbor status?
robbie
"Paul Stewart"
<pstewart at nexicom
group.net>
To
Sent by: cisco-nsp at puck.nether.net
cisco-nsp-bounces
cc
@puck.nether.net
Subject
[c-nsp] Link Problem - 6509 to
GSR
10/23/2006 09:23
AM
I have a weird problem between a 6509 and our GSR router:
On the 6509:
interface GigabitEthernet9/6
switchport
switchport access vlan 30
switchport mode access
no ip address
speed nonegotiate
interface Vlan30
ip address 10.250.250.1 255.255.255.0
core1-rtr-mb#sh interfaces gigabitEthernet 9/6
GigabitEthernet9/6 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 0009.127a.2ce5 (bia
0009.127a.2ce5)
Description:
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is SX
input flow-control is off, output flow-control is on
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 2d23h, output 00:00:33, output hang never
Last clearing of "show interface" counters 2d23h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
378 packets input, 29833 bytes, 0 no buffer
Received 12 broadcasts (2 multicasts)
35 runts, 0 giants, 0 throttles
35 input errors, 0 CRC, 35 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
1208 packets output, 98465 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
core1-rtr-mb#sh interfaces vlan30
Vlan30 is up, line protocol is up
Hardware is EtherSVI, address is 0004.defd.f40a (bia 0004.defd.f40a)
Internet address is 10.250.250.1/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:02:07, output 00:00:48, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
L2 Switched: ucast: 477 pkt, 37840 bytes - mcast: 4 pkt, 290 bytes
L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 0 pkt, 0 bytes
1 packets input, 64 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
15 packets output, 1588 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
core1-rtr-mb#ping 10.250.250.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.250.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
On the GSR:
interface GigabitEthernet5/0
ip address 10.250.250.2 255.255.255.0
no ip directed-broadcast
no negotiation auto
core1-gsr-mb#sh interfaces gigabitEthernet 5/0 GigabitEthernet5/0 is up,
line protocol is up
Hardware is GigMac GigabitEthernet, address is 0050.e28a.bda9 (bia
0050.e28a.bda9)
Internet address is 10.250.250.2/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, rely 255/255, load 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is force-up, media type is SX
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d23h
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
82 packets input, 18738 bytes, 0 no buffer
Received 75 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 858 multicast, 0 pause input
403 packets output, 27276 bytes, 0 underruns
Transmitted 282 broadcasts
0 output errors, 0 collisions, 8 interface resets
0 babbles, 0 late collision, 0 deferred
3 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
core1-gsr-mb#ping 10.250.250.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.250.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
BUT.. here's the problem:
core1-gsr-mb#ping 10.250.250.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.250.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
This seems like such an easy task... but it's got me baffled why I can't
pass traffic over this link. I've tried two different GigE ports on
both sides.... re-created the VLAN etc. and even made the interface on
the 6509 a layer3 interface directly (without binding a vlan to it) and
can't pass traffic... it's almost gotta be physical but wondering if
anyone has any ideas?? I'm kinda feeling dumb at this point in time when
we have other links between these two devices and they work fine....
Thanks in advance.
Paul
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list