[c-nsp] ASR9k LC fib programming problem

Šturmankin Miroslav miroslav.sturmankin at swan.sk
Fri Jul 14 04:59:30 EDT 2017


Hi Jason,

thanks for the hint, seems to be the case of version mis-match on the LC:

RP/0/RSP0/CPU0:a9k6#sh cef vrf inet summary location 0/1/CPU0 | in CEF
Fri Jul 14 10:40:34.125 CEST
IP CEF with switching (Table Version 0) for node0_1_CPU0
  0 CEF route update drops, 371317788 revisions of existing leaves
  16 CEF route update drops due to version mis-match

RP/0/RSP0/CPU0:a9k6#conf t
Fri Jul 14 10:40:57.274 CEST
RP/0/RSP0/CPU0:a9k6(config)#router static 
RP/0/RSP0/CPU0:a9k6(config-static)#vrf inet
RP/0/RSP0/CPU0:a9k6(config-static-vrf)#address-family ipv4 unicast 
RP/0/RSP0/CPU0:a9k6(config-static-vrf-afi)#x.x.x.0/24 b.b.b.2 description customerA_static
RP/0/RSP0/CPU0:a9k6(config-static-vrf-afi)#commit

RP/0/RSP0/CPU0:a9k6#sh cef vrf inet summary location 0/1/CPU0 | in CEF
Fri Jul 14 10:42:01.584 CEST
IP CEF with switching (Table Version 0) for node0_1_CPU0
  0 CEF route update drops, 371318426 revisions of existing leaves
  17 CEF route update drops due to version mis-match

RP/0/RSP0/CPU0:a9k6#conf t
Fri Jul 14 10:42:17.034 CEST
RP/0/RSP0/CPU0:a9k6(config-static)#vrf inet                                               
RP/0/RSP0/CPU0:a9k6(config-static-vrf)#address-family ipv4 unicast                            
RP/0/RSP0/CPU0:a9k6(config-static-vrf-afi)#no x.x.x.0/24 b.b.b.2 description customerA _static
RP/0/RSP0/CPU0:a9k6(config-static-vrf-afi)#commit

RP/0/RSP0/CPU0:a9k6#sh cef vrf inet summary location 0/1/CPU0 | in CEF
Fri Jul 14 10:42:35.922 CEST
IP CEF with switching (Table Version 0) for node0_1_CPU0
  0 CEF route update drops, 371319294 revisions of existing leaves
  18 CEF route update drops due to version mis-match

Shouldn't this be fixed in 4.3.4 already? Also, there is no record in log about route-update skipped as mentioned in the bug. And the number is increasing only when manipulating this particular route (the numner 18 should be about right), other updates get programmed ok (I just tried it with a different prefix and cef on the same LC was correct). I don't believe there has been over 2 billion route updates for this particular prefix :)

I guess I will have to find a way how to force the update over version mis-match. If anyone can share some info regarding this, please advise. I'll update the list once I get this any further.

Thank you all.

Regards/S pozdravom,

Miro Sturmankin

From: Jason Lixfeld [mailto:jason at lixfeld.ca] 
Sent: Friday, July 14, 2017 9:01 AM
To: Šturmankin Miroslav
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] ASR9k LC fib programming problem

Hi,

We have a few cases of this right now pending a maintenance window to resolve:

https://bst.cloudapps.cisco.com/bugsearch/bug/cscui93977
Sent from my iPhone

On Jul 13, 2017, at 5:02 PM, Šturmankin Miroslav <miroslav.sturmankin at swan.sk> wrote:
Hello everyone,

I encountered a problem with traffic being forwarded to the old next-hop (no lc hw reprogramed for the new bgp next-hop) after converting route for a prefix from static to bgp.

router specifications:
ASR9006 IOS XR 4.3.4 SP4
A9K-RSP-4G
A9K-MOD160-TR + A9K-MPA-8X10GE & A9K-MPA-4X10GE

I have a bundle-ether interface spanning over two 10G links on both MPAs, which is terminating customer traffic. I have a customer with two links terminated on separate subinterfaces of the bundle and had a /24 prefix routed to a p2p /30 next-hop over the first interface.

int bundle-ether1.10
description [stat] customerA
vrf inet
ipv4 address a.a.a.1/30
encapsulation dot1q 10

router static
vrf inet
 address-family ipv4 unicast
  x.x.x.0/24 a.a.a.2 description customerA_static

The customer wanted to move the prefix to the second interface and convert static to ebgp advertisement.

int bundle-ether1.20
description [bgp] customerA
vrf inet
ipv4 address b.b.b.1/30
encapsulation dot1q 20

router bgp 1
vrf inet
 neighbor b.b.b.2
  remote-as 2
  description customerA_bgp_peer
  address-family ipv4 unicast
   route-policy RP_ACCEPT_CUSTOMERA_PREFIXES in

The ebgp session was already established advertising other prefixes a while ago and he advertised the new prefix over bgp. I then removed the static route towards next-hop a.a.a.2 and the bgp route got installed into the routing table. However, the traffic for x.x.x.0/24 was still forwarded from the router via the first interface be1.10, even though routing table was pointing towards next-hop b.b.b.2 via be1.20. After some troubleshooting with extended ping and record route I confirmed that the traffic was leaving my router via be1.10 and returned via be1.20. So I checked fib for a problem and found out, that the fib on LC is still pointing to a.a.a.2 next-hop:

RP/0/RSP0/CPU0:a9k6#sh route vrf inet x.x.x.0/24
Thu Jul 13 16:36:47.547 CEST

Routing entry for x.x.x.0/24
 Known via "bgp 1", distance 20, metric 0
 Tag 111, type external
 Installed Jul 12 11:35:40.360 for 1d05h
 Routing Descriptor Blocks
   b.b.b.2, from b.b.b.2, BGP external
     Route metric is 0
 No advertising protos.

RP/0/RSP0/CPU0:a9k6#sh bgp vrf inet x.x.x.0/24
Thu Jul 13 16:38:09.485 CEST
BGP routing table entry for x.x.x.0/24, Route Distinguisher: bla:bla
Versions:
 Process           bRIB/RIB  SendTblVer
 Speaker         2625784714  2625784714
   Local Label: 1
Last Modified: May 27 18:14:36.243 for 2y06w
Paths: (1 available, best #1)
 2
   b.b.b.2 from b.b.b.2 (b.b.b.2)
     Origin IGP, localpref 600, valid, external, best, group-best, import-candidate
     Received Path ID 0, Local Path ID 1, version 2625784714
     Community: bla bla
     Extended community: RT:bla:bla

RP/0/RSP0/CPU0:a9k6#sh cef vrf inet x.x.x.0/24
Thu Jul 13 16:42:57.056 CEST
x.x.x.0/24, version 6698698982, internal 0x4000001 (ptr 0xb0d34a14) [1], 0x0 (0x0), 0x0 (0x0)
Updated Jul 12 11:35:40.369
Prefix Len 24, traffic index 0, precedence n/a, priority 3
  via b.b.b.2, 7 dependencies, recursive, bgp-ext [flags 0x6020]
   path-idx 0 [0xb2200544 0x0]
   next hop b.b.b.2 via b.b.b.2/32

RP/0/RSP0/CPU0:a9k6#sh cef vrf inet x.x.x.0/24 hardware egress 
Thu Jul 13 16:44:01.928 CEST
x.x.x.0/24, version 6698698982, internal 0x4000001 (ptr 0xb0d34a14) [1], 0x0 (0x0), 0x0 (0x0)
Updated Jul 12 11:35:40.369
Prefix Len 24, traffic index 0, precedence n/a, priority 3
  via b.b.b.2, 7 dependencies, recursive, bgp-ext [flags 0x6020]
   path-idx 0 [0xb2200544 0x0]
   next hop b.b.b.2 via b.b.b.2/32

RP/0/RSP0/CPU0:a9k6#sh cef vrf inet x.x.x.0/24 location 0/1/CPU0
Thu Jul 13 16:48:55.236 CEST
x.x.x.0/24, version 3115832151, internal 0x4000001 (ptr 0x8a147d64) [1], 0x0 (0x0), 0x0 (0x0)
Updated Nov  9 00:24:24.757
Prefix Len 24, traffic index 0, precedence n/a, priority 3
  via a.a.a.2, 3 dependencies, recursive [flags 0x0]
   path-idx 0 [0x90220fe4 0x0]
   next hop a.a.a.2 via a.a.a.2/32

RP/0/RSP0/CPU0:a9k6#sh cef vrf inet x.x.x.0/24 hardware egress location 0/1/CPU0
Thu Jul 13 16:50:58.198 CEST
x.x.x.0/24, version 3115832151, internal 0x4000001 (ptr 0x8a147d64) [1], 0x0 (0x0), 0x0 (0x0)
Updated Nov  9 00:24:24.757
Prefix Len 24, traffic index 0, precedence n/a, priority 3
  via a.a.a.2, 3 dependencies, recursive [flags 0x0]
   path-idx 0 [0x90220fe4 0x0]
   next hop a.a.a.2 via a.a.a.2/32
LEAF - HAL pd context : 
sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0
Leaf H/W Result:

   Physical Result: 0x11680600 (LE) 

   Raw Data0: 0x71800003 00000000 00000000 00000000
   Raw Data1: 0x4e030000 00000000 00180000 00000000
   leaf_resolve_control_byte0
                   reserved: 0                           match: 1                   valid: 1
             txadj_internal: 0
          recursive: 1
                     rec_fs: 1
   leaf_resolve_control_byte1
                        fwd: 1             default_rte: 0
             dc_rte: 0
                ifib_lookup: 0
        fast_switch: 0
       feature_lkup: 0
                   igp_pref: 0                   non_recursive: 0
   leaf_resolve_control_byte2
          more_features: 0
           bgp_pa_valid: 0
            fast_switch: 0
              ecmp_size: 3
   recursive_fwd_entry
                    ldi_ptr: 0x4e0300 (LE)

       LSP Array union:
       BGP Output label:
               label_msb[0]: 0                    label_lsb[0]: 0
                     exp[0]: 0                          eos[0]: 0
               label_msb[1]: 0                    label_lsb[1]: 0
                     exp[1]: 0                          eos[1]: 0
               label_msb[2]: 0                    label_lsb[2]: 0
                     exp[2]: 0                          eos[2]: 0
               label_msb[3]: 0                    label_lsb[3]: 0
                     exp[3]: 0                          eos[3]: 0

              lsp_array_ptr: 0x0
       bgp_policy_accounting: 0
                  as_number: 0
              prefix_length: 18
         QPPB Prec: 0            QPPB Prec_valid: 0
       QPPB QOS Group: 0            QPPB QOS Group_valid: 0
REC-SHLDI HAL PD context :
collapse_bwalk_required:0, load_shared_lb:0

rLDI eng ctx:
   flags: 0x1, rldi_table_idx: 0x34e, num_entries: 0x1, urpf_ptr: 0x0

rLDI HW data for path 0 [index: 0x34e (BE)] (common to all NPs):

   Raw Data0: 0x11044004 737c0000 d94b5b22 00000000
   ldi_resolve_control_byte0:
                      match: 1                   valid: 1

   ldi_resolve_control_byte1:
                   ext_lspa: 0                 leaf_ip: 1
                  leaf_mpls: 0

   ldi_resolve_control_byte2:
                 first_rldi: 1
               label_offset: 4
         label_block_offset: 0

   Other fields:
                   leaf_ptr: 0x737c00(LE)            bgp_next_hop: 0xaaa2
NextHopPrefix:a.a.a.2/32

Please use show cef  or show mpls forwarding command again
with nexthop prefix specified for nexthop hardware details

I have tried several things to reprogram the LC but no success, the record is still stuck into the LC HW:
1) added back the static route towards a.a.a.2 and removed it
2) added a static route towards b.b.b.2
3) cleared route ipv4 unicast
4) removed the prefix x.x.x.0/24 from routing table
5) shut down the be1.10 interface

Has anybody encoutered enythig similar on their a9k boxes and how did you please solve the issue (other than LC reload)? Thank you.

Regards/S pozdravom,

Miroslav Sturmankin
Oddelenie sietovej architektury
SWAN, a.s.
Borska 6, 841 04 Bratislava
miroslav.sturmankin at swan.sk
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


More information about the cisco-nsp mailing list