[c-nsp] issue on old but running device
james list
jameslist72 at gmail.com
Thu Jan 30 16:08:00 UTC 2025
Hi Brian
thanks for responding.
I've already done some actions, reboot did not help, the point is that the
same C3925 getting the same feeds in other location doesn't have the same
behaviour.
One difference: 3925 having the issue is connected 1 Gbs to a C9407 from
the market data feed broker, while the 3925 not having the issue is
connected to an old C4948...
Unfortunately I have already done some of the actions, not tried rate-limit
to log the exceed:
1) reboot doesn't helped
2) buffer miss doesn't increase a lot
#sh buffer
Buffer elements:
603 in free list
96825593 hits, 0 misses, 1242 created
Public buffer pools:
Small buffers, 104 bytes (total 1024, permanent 1024):
1016 in free list (100 min, 2048 max allowed)
39265134 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 1024, permanent 1024):
1024 in free list (100 min, 2048 max allowed)
12407239 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 512, permanent 512):
512 in free list (100 min, 1024 max allowed)
25923483 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 150, permanent 150, peak 151 @ 7w0d):
150 in free list (30 min, 300 max allowed)
78341 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 1 @ 7w0d):
1 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 486 trims, 487 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 5, permanent 0, peak 8 @ 7w0d):
5 in free list (4 min, 10 max allowed)
234041 hits, 972 misses, 2917 trims, 2922 created
0 failures (0 no memory)
Interface buffer pools:
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 486 trims, 487 created
0 failures (0 no memory)
Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @
7w0d):
101 in free list (100 min, 100 max allowed)
0 hits, 0 misses
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 7w0d):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 486 trims, 487 created
0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @
7w0d):
101 in free list (132 min, 132 max allowed)
318 hits, 0 misses
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 7w0d):
26 in free list (25 min, 50 max allowed)
0 hits, 0 misses, 486 trims, 487 created
0 failures (0 no memory)
IPC buffers, 4096 bytes (total 2, permanent 2):
1 in free list (1 min, 8 max allowed)
1 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 7w0d):
3 in free list (2 min, 4 max allowed)
0 hits, 0 misses, 486 trims, 487 created
0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 7w0d):
2 in free list (1 min, 2 max allowed)
0 hits, 0 misses, 486 trims, 487 created
0 failures (0 no memory)
IPC Medium buffers, 16384 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 7w0d):
17 in free list (16 min, 16 max allowed)
0 hits, 0 misses, 78809 trims, 78810 created
0 failures (0 no memory)
Header pools:
Header buffers, 0 bytes (total 8705, permanent 8192, peak 8705 @ 7w0d):
513 in free list (512 min, 16384 max allowed)
8021 hits, 171 misses, 972 trims, 1485 created
0 failures (0 no memory)
8192 max cache size, 8192 in cache
13200 hits in cache, 0 misses in cache
Particle Clones:
1024 clones, 19303 hits, 0 misses
Public particle pools:
F/S buffers, 256 bytes (total 3072, permanent 3072):
1024 in free list (512 min, 4096 max allowed)
2048 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
2048 max cache size, 2048 in cache
24526 hits in cache, 0 misses in cache
Normal buffers, 1548 bytes (total 3840, permanent 3840):
3712 in free list (128 min, 4096 max allowed)
128 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache
CRYPTO_HW buffers, 1600 bytes (total 12288, permanent 12288):
0 in free list (0 min, 12288 max allowed)
12288 hits, 0 misses
12288 max cache size, 12288 in cache
234482 hits in cache, 0 misses in cache
Private particle pools:
HQF Particle buffers, 0 bytes (total 2000, permanent 2000):
2000 in free list (500 min, 2000 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
IDS SM buffers, 240 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
128 hits, 0 fallbacks
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache
GigabitEthernet0/0 buffers, 2304 bytes (total 2561, permanent 2048):
513 in free list (512 min, 6144 max allowed)
1877 hits, 171 misses, 972 trims, 1485 created
0 failures (0 no memory)
2048 max cache size, 1536 in cache
31641103 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/1 buffers, 2304 bytes (total 2561, permanent 2048):
513 in free list (512 min, 6144 max allowed)
1877 hits, 171 misses, 972 trims, 1485 created
0 failures (0 no memory)
2048 max cache size, 1536 in cache
2796374359 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/2 buffers, 2304 bytes (total 2561, permanent 2048):
513 in free list (512 min, 6144 max allowed)
1877 hits, 171 misses, 972 trims, 1485 created
0 failures (0 no memory)
2048 max cache size, 1536 in cache
1095825533 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/3 buffers, 2304 bytes (total 2561, permanent 2048):
513 in free list (512 min, 6144 max allowed)
1877 hits, 171 misses, 972 trims, 1485 created
0 failures (0 no memory)
2048 max cache size, 1536 in cache
512 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
3) show controller
# sh controllers gigabitEthernet 0/2
Interface GigabitEthernet0/2 (idb 0x22B2F87C)
Hardware is iGbE
Register base address: 0xFD120000
PCI REV: A1
network link is up
loopback type is none
= Media RJ45 =
Media Change Count: 1
MAC registers:
CTRL =0x000C1A41, STATUS=0x80080783, CTRL_X=0x12800480, IMS =0x00000091
EECD =0x00000218, EERD =0x00000000 ETT =0x00000000, TXDMAC=0x00000000
RCTL =0x00408032, RDBAL =0x10092000, RDBAH =0x00000000, RDLEN =0x00002000
RDH =0x00000000, RDT =0x000001FF, RDTR =0x0000130D, RLPML =0x000005F4
RFCTL =0x00800000
TCTL =0xA40400FA, TDBAL =0x10095000, TDBAH =0x00000000, TDLEN =0x00004000
TDH =0x00000178, TDT =0x00000178, TIPG =0x00701808
TDWAH =0x00000000, TDWAL =0x0FA13803
FCRTH =0x00000000, FCRTL =0x00000000
FCAH =0x00000100, FCAL =0x00C28001, FCT =0x00008808, FCTTV =0x00000000
RDFH =0x40021A70, RDFT =0x00001A24, RDFPC =0x00000000
TDFH =0x4002130E, TDFT =0x0000130C, TDFPC =0x00000000
RDFHS =0x00001AB0, RDFTS =0x00001AB0, RXDCTL=0x02010000
TDFHS =0x0000130E, TDFTS =0x0000130E, TXDCTL=0x02000000
RXCSUM=0x00000300, TIDV=0x00000020
PCS_CFG =0x00000008, PCS_LCTL=0x02110000, PCS_LSTS =0x0061001D
PCS_ANADV =0x00000020, PCS_LPAB=0x0000D800, PCS_NPTX =0x00000000
PCS_LPABNP=0x00000000, I2CCMD =0x00000000, I2CPARAMS=0x00001746
Rxdescriptor addr=10532100 len=48310437, stat=00000000
Txdescriptor addr=0F8400FC cmd/len=0B00004E, stat=00000000
RX is enabled, TX is enabled
Device status = full-duplex, link up
PCI width = 32, PCI66=0, PCIX=0, PCIXSPD is 50-66MHz Device Speed =
1000Mbps
PCI configuration registers:
bus_no=2, device_no=0, subdevice 0
DeviceID=0x10C9, VendorID=0x8086, Command=0x0146, Status=0x0010
Class=0x02/0x00/0x00, Revision=0x01, LatencyTimer=0x00, CacheLineSize=0x10
BaseAddr0=0xFD120000, BaseAddr1=0x00000000, MaxLat=0x00,MinGnt=0x00
SubsysDeviceID=0x0000, SubsysVendorID=0x8086
Interrupt Line 0x0B, Pin 0x01
Cap_Ptr=0x00000040 Retry/TRDY Timeout=0x48235001
PMC=0x00000000 PMCSR=0x00000003
DEVCTRL=0x00090800
Internal Driver Information:
lc_ip_turbo_fs=0x0, ip_routecache=0x11(dfs=0/mdfs=0)
i82576_ds=0x22B30BB8, registers=0xFD120000
mci_txcount 1024 rx cache size=2048, rx cache end=1536, rx_nobuffer=0
max_mtu=1528
ring sizes: RX=512, TX=1024
rxring=0x10092000, shadow=0x22B3165C, head=0, rx_buf_size=2048
txring=0x10095000, shadow=0x22B31E90, sw_tdh=376, tail=376
chip_state=2, pci_rev=1
congested = 0 tx_count=0, tx_ring_max=1024, tx_limited=0 (cur ring size
1024)
rx_overrun=58854, rx_seq=0, rx_no_enp=0, rx_discard=0
throttled=0, enabled=0, disabled=0, bypassed=0
reset=4(init=1, check=0, restart=3, pci=0), auto_restart=8
link_reset=0, tx_carrier_loss=4, fatal_tx_err=0
wait_for_last_tdt=0, rx_stuck=0
tx_stuck=0, rx_max_spin=1, rr_port_next=1
Statistics:
CRC error 0 Symbol error 0
Missed Packets 58854 Excessive Coll 0
Multiple Coll 0 Length Error 0
Late Coll 0 Collision 0
Defer 0 Receive Length 0
XON RX 0
XON TX 0 XOFF RX 0
XOFF TX 0 FC RX Unsupport 0
Packet RX (64) 0 Packet RX (127) 5486324
Packet RX (255) 2568423 Packet RX (511) 764976
Packet RX (1023) 598611 Packet RX (1522) 3299107
Good Packet RX 12717441 Broadcast RX 0
Multicast RX 12688463 Good Packet TX 44078
Good Octets RX.H 1 Good Octets RX.L 1954032566
Good Octets TX.H 0 Good Octets TX.L 5414177
RX No Buff 9421 RX Undersize 0
RX Fragment 0 RX Oversize 0
RX Octets High 1 RX Octets Low 1961371318
TX Octets High 0 TX Octets Low 5414177
TX Packet 44078 RX Packet 12776868
TX Broadcast 0 TX Multicast 5968
Packet TX (64) 3912 Packet TX (127) 32710
Packet TX (255) 2567 Packet TX (511) 4889
Packet TX (1023) 0 Packet TX (1522) 0
TX No CSR 0
RX Error Count 0 TCP Segmentation 0
Host Trasmit Discarded packets by MAC Count 0
Interrupt Assertion Count 5809488
Receive Packets to Host Count 12717440
Host Good Packets Transmitted Count 44078
PHY REGISTERS:
Display for CRC check for page:6
PACKET_GENERATION reg_addr: 16: 0x0010
CRC COUNTERS reg_addr: 17: 0xFF00
page number:0
Copper Contrl @ reg_addr: 0: 0x1140
Copper Status @ reg_addr: 1: 0x796D
PHY Idf1 @ reg_addr: 2: 0x0141
PHY Idf2 @ reg_addr: 3: 0x0C97
Autoneg Adv @ reg_addr: 4: 0x0C01
Autoneg Lnk Ptr abty @ reg_addr: 5: 0xC001
Autoneg Exp @ reg_addr: 6: 0x000D
Autoneg Nx Pg Tx @ reg_addr: 7: 0x2801
Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x4D26
1000B-T Ctr @ reg_addr: 9: 0x0E00
1000B-T St @ reg_addr: 10: 0x3800
Copper Ext St @ reg_addr: 15: 0x3000
Copper Specific Ctl1 @ reg_addr: 16: 0x6060
Copper Specific St1 @ reg_addr: 17: 0xAC08
Copper Interrupt En @ reg_addr: 18: 0x0000
Copper Specific St2 @ reg_addr: 19: 0x0000
Copper Rcv Er Ctr @ reg_addr: 21: 0x0000
Copper Specific Ctl2 @ reg_addr: 26: 0x8040
page number:1
Fiber Contrl @ reg_addr: 0: 0x1140
Fiber Status @ reg_addr: 1: 0x794D
PHY Idf1 @ reg_addr: 2: 0x0141
PHY Idf2 @ reg_addr: 3: 0x0C97
Autoneg Adv @ reg_addr: 4: 0x0060
Autoneg Lnk Ptr abty @ reg_addr: 5: 0x0000
Autoneg Exp @ reg_addr: 6: 0x0004
Autoneg Nx Pg Tx @ reg_addr: 7: 0x2001
Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x0000
Fiber Ext St @ reg_addr: 15: 0x3000
Fiber Specific Ctl1 @ reg_addr: 16: 0x0000
Fiber Specific St1 @ reg_addr: 17: 0x0018
Fiber Interrupt En @ reg_addr: 18: 0x0000
Fiber Specific St2 @ reg_addr: 19: 0x0000
Fiber Rcv Er Ctr @ reg_addr: 21: 0x0000
Fiber Specific Ctl2 @ reg_addr: 26: 0x8085
page number:2
MAC Contrl @ reg_addr: 0: 0x1040
PHY Idf1 @ reg_addr: 2: 0x0141
PHY Idf2 @ reg_addr: 3: 0x0C97
MAC Specific Ctl1 @ reg_addr: 16: 0x0288
MAC Specific St1 @ reg_addr: 17: 0x0440
MAC Interrupt En @ reg_addr: 18: 0x0000
MAC Specific St2 @ reg_addr: 19: 0x0000
MAC Specific Ctl2 @ reg_addr: 26: 0x0005
HW addr filter: 0x22B33EC4
Chip address filter[0] 80000293.16032B44
Chip address filter[1] 80000E00.00C28001
Chip address filter[2] 8000CCCC.CC0C0001
Chip address filter[3] 80000700.00C28001
Chip address filter[4] 80000200.00C28001
Chip address filter[5] 03FFFFFF.005E0001
Chip address filter[6] 03FFFFFF.005E0001
Chip address filter[7] 03FFFFFF.005E0001
Chip address filter[8] 03FFFFFF.005E0001
Chip address filter[9] 03FFFFFF.00000000
Chip address filter[10] 03FFFFFF.00000000
Chip address filter[11] 03FFFFFF.00000000
Chip address filter[12] 03FFFFFF.00000000
Chip address filter[13] 03FFFFFF.00000000
Chip address filter[14] 03FFFFFF.00000000
Chip address filter[15] 03FFFFFF.00000000
Entry= 0: Addr=442B.0316.9302
Entry= 1: Addr=0180.C200.000E
Entry= 2: Addr=0100.0CCC.CCCC
Entry= 3: Addr=0180.C200.0007
Entry= 4: Addr=0180.C200.0002
(All other entries are empty)
Software MAC address filter(hash:length/addr/mask/hits)
-------------------------------------------------------
0x00: 0 ffff.ffff.ffff 0000.0000.0000 5
0x01: 0 442b.0316.9302 0000.0000.0000 1035374
0xC0: 0 0100.0ccc.cccc 0000.0000.0000 12399
0xC0: 1 0180.c200.0002 0000.0000.0000 0
0xC5: 0 0180.c200.0007 0000.0000.0000 0
0xCC: 0 0180.c200.000e 0000.0000.0000 2
0x00: 0 0100.5e7f.ffff 0000.007f.ffff 1094516179
Software filtered frames: 261439
unicast overflow mode off
multicast overflow mode on
Promiscuous mode off
Il giorno gio 30 gen 2025 alle ore 16:41 Brian Turnbow <
brian.turnbow at cdlan.it> ha scritto:
> Hi James
>
> You can use
> sh buffers and sh controller g0/2 for more info and then try and tune
> buffers, but this doesn't help a lot usually.
> Another thing to check would be for microbursts using the old rate limit
> trick setting a policer with exceed action transmit
> Then you can check to see if you are overflowing therate limit bucket as
> sign of microburst. Like maybe due to speed steps 10G/1G.
> Overruns are a sign that the router can't handle the traffic coming in and
> may be due to traffic levels or traffic types that take longer to process.
> Flow control could help but I see unsupported listed on the interface and
> admit I am not well versed on the 3925 as a platform
> Have you tried a reboot ? I have seen routers get out of whack that
> magically return to form after a reboot..
>
>
>
> Brian Turnbow
>
>
>
> Il giorno gio 30 gen 2025 alle ore 14:41 james list via cisco-nsp <
> cisco-nsp at puck.nether.net> ha scritto:
>
>> Dear expert
>> I'm trying to get any useful insight and I can also understand not polite
>> comments...
>>
>> We've a customer which is experiencing multicast gaps on feed A and we've
>> identified an old C3925-SPE200 in a remote location (hence difficult to be
>> replaced in a manner) which is experiencing input errors/overrun also on
>> low bandwidth and packet rate.
>> The customer has the same kind of device in another remote location
>> receiving the same multicast feed (feed B) without issues.
>> I think these gaps are due to packet dropped ... customer has also tried
>> to
>> move interface from g0/1 to g0/2 but no improvement has obtained.
>>
>> Any idea what could be the reason ?
>>
>> Thanks in advance
>> James
>>
>> xxx>sh inventory
>> NAME: "CISCO3925-CHASSIS", DESCR: "CISCO3925-CHASSIS"
>> PID: CISCO3925-CHASSIS , VID: V02, SN: FCZxxx
>>
>> NAME: "Cisco Services Performance Engine 200 for Cisco 3900 ISR on Slot
>> 0",
>> DESCR: "Cisco Services Performance Engine 200 for Cisco 3900 ISR"
>> PID: C3900-SPE200/K9 , VID: V04 , SN: FOC160320B7xxx
>>
>> NAME: "C3900 AC Power Supply 1", DESCR: "C3900 AC Power Supply 1"
>> PID: PWR-3900-AC , VID: V03, SN: SNIxxx
>>
>> NAME: "C3900 AC Power Supply 2", DESCR: "C3900 AC Power Supply 2"
>> PID: PWR-3900-AC , VID: V03 , SN: QCSxxx
>>
>>
>> xxx>sh int g0/2
>> GigabitEthernet0/2 is up, line protocol is up
>> Hardware is iGbE, address is 442b.0316.9302 (bia 442b.0316.9302)
>> Description: To Market
>> MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
>> reliability 255/255, txload 1/255, rxload 2/255
>> Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
>> Keepalive set (10 sec)
>> Full Duplex, 1Gbps, media type is RJ45
>> output flow-control is unsupported, input flow-control is unsupported
>> ARP type: ARPA, ARP Timeout 04:00:00
>> Last input 00:00:00, output 00:00:00, output hang never
>> Last clearing of "show interface" counters 01:18:46
>> Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
>> Queueing strategy: fifo
>> Output queue: 0/40 (size/max)
>> 30 second input rate 9004000 bits/sec, 2628 packets/sec
>> 30 second output rate 9000 bits/sec, 7 packets/sec
>> 7936911 packets input, 4224448667 bytes, 0 no buffer
>> Received 7914597 broadcasts (7218973 IP multicasts)
>> 0 runts, 0 giants, 0 throttles
>> 56932 input errors, 0 CRC, 0 frame, 56932 overrun, 0 ignored
>> 0 watchdog, 7914597 multicast, 0 pause input
>> 30904 packets output, 2951099 bytes, 0 underruns
>> 0 output errors, 0 collisions, 0 interface resets
>> 158 unknown protocol drops
>> 0 babbles, 0 late collision, 0 deferred
>> 0 lost carrier, 0 no carrier, 0 pause output
>> 0 output buffer failures, 0 output buffers swapped out
>>
>>
>> interface GigabitEthernet0/2
>> description To Market
>> no ip address
>> load-interval 30
>> duplex full
>> speed 1000
>> hold-queue 2000 in
>> !
>> interface GigabitEthernet0/2.90
>> description VLAN multicast
>> encapsulation dot1Q 90
>> ip address xx.xx.221.1 255.255.255.0
>> ip pim query-interval 5
>> ip pim bsr-border
>> ip pim dense-mode proxy-register list xx
>> ip multicast boundary xx
>> ip nat outside
>> ip virtual-reassembly in
>> !
>> interface GigabitEthernet0/2.1683
>> description To market flex vpn
>> encapsulation dot1Q 1683
>> ip address 10.xx.xx.85 255.255.255.252
>> ip access-group CRYPTO-xxx
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>
More information about the cisco-nsp
mailing list