[j-nsp] bgp routes not passing traffic between routers, traffic dropped

[Bill Petrisko]

I am having a strange problem with certain BGP routes
not being forwarded correctly to a neighboring router.

It started yesterday with one /17 prefix not working,
and today we have another handful of /20, /23, /24's 
all not forwarding.

Router 1 is our colo router   M20 6.2R2.2
Router 2 is our core router   M20 5.6R2.4

Path should be server->router 1->router 2->internet/core

>From a local server (same from router 1 itself)-

# traceroute 66.173.240.20 
traceroute to 66.173.240.20 (66.173.240.20), 30 hops max, 38 byte packets
  1  xx.xx.xx.xx  (router 1)
  2  * *   (router 2)

>From router 1-

The route in BGP looks good:

bill at router1> show route 66.173.240.20                                              
inet.0: 150815 destinations, 150824 routes (150802 active, 15 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

66.173.240.0/23    *[BGP/170] 5d 18:52:15, MED 0, localpref 100, from 69.28.139.222
                      AS path: 3549 701 16810 23241 I
                      to 69.28.139.249 via ge-1/1/0.0
                    > to 69.28.139.253 via ge-2/1/0.0

The next-hops look good:

bill at router1> show route 69.28.139.249  

inet.0: 150794 destinations, 150803 routes (150792 active, 4 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

69.28.139.248/30   *[Direct/0] 2w1d 10:53:27
                    > via ge-1/1/0.0

bill at router1> show route 69.28.139.253    

inet.0: 150794 destinations, 150803 routes (150792 active, 4 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

69.28.139.252/30   *[Direct/0] 1d 21:21:30
                    > via ge-2/1/0.0


bill at router1> show route forwarding-table destination 66.173.240.20 
Routing table: inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
66.173.240.0/23    user     0                    indr   976 48557
                                                 ulst  1335   522
                              69.28.139.249      ucst   421     3 ge-1/1/0.0  
                              69.28.139.253      ucst   422     3 ge-2/1/0.0  


bill at router1> show route resolution 66.173.240.0/23 
Table inet.3 Nodes 0
66.173.240.0/23 Originating RIB: inet.0
  Metric: 30  Node path count: 1
  Indirect nexthops: 1
        Protocol Nexthop: 208.50.254.245 Metric: 30
        Indirect nexthop: 8763180 976
        Indirect path forwarding nexthops: 2
                Nexthop: 69.28.139.249 via ge-1/1/0.0
                Nexthop: 69.28.139.253 via ge-2/1/0.0
Table inet.0 Nodes 150801
66.173.240.0/23 Originating RIB: inet.0
  Metric: 30  Node path count: 1
  Indirect nexthops: 1
        Protocol Nexthop: 208.50.254.245 Metric: 30
        Indirect nexthop: 8763180 976
        Indirect path forwarding nexthops: 2
                Nexthop: 69.28.139.249 via ge-1/1/0.0
                Nexthop: 69.28.139.253 via ge-2/1/0.0
Table inet.2 Nodes 150801
66.173.240.0/23 Originating RIB: inet.0
  Metric: 30  Node path count: 1
  Indirect nexthops: 1
        Protocol Nexthop: 208.50.254.245 Metric: 30
        Indirect nexthop: 8763180 976
        Indirect path forwarding nexthops: 2
                Nexthop: 69.28.139.249 via ge-1/1/0.0
                Nexthop: 69.28.139.253 via ge-2/1/0.0


Putting a filter on the router2 interface shows the packets are
being received for this prefix:

bill at router2-re0> show configuration firewall filter trackit 
term log {
    from {
        destination-address {
            66.173.240.20/32;
        }
    }
    then {
        count trackit;
        log;
        accept;
    }
}
term default {
    then accept;
}

(applied only to router1->router2 ckt #1)

bill at router2-re0> show firewall filter trackit                  
Filter: trackit                                                
Counters:
Name                                                Bytes              Packets
trackit                                             51900                   67

(applied only to router1->router2 ckt #1)
bill at router2-re0> show firewall filter trackit    
Filter: trackit                                                
Counters:
Name                                                Bytes              Packets
trackit                                             44016                   42



Route works fine from router 2-

bill at router2-re0> traceroute 66.173.240.20            
traceroute to 66.173.240.20 (66.173.240.20), 30 hops max, 40 byte packets
 1  ge-3-2-0.ar5.phx1.gblx.net (64.208.170.253)  1.040 ms  0.946 ms  11.765 ms
 2  so1-0-0-2488M.ar1.LAX2.gblx.net (67.17.67.169)  16.094 ms so0-0-0-2488M.ar1.LAX2.gblx.net (67.17.72.106)  10.722 ms  10.569 ms
 3  208.51.134.2 (208.51.134.2)  11.994 ms  11.863 ms  11.874 ms
 4  0.so-0-3-0.XL1.LAX9.ALTER.NET (152.63.115.2)  12.005 ms  12.201 ms  11.991 ms
 5  0.so-0-0-0.TL1.LAX9.ALTER.NET (152.63.115.138)  12.252 ms  12.402 ms  12.123 ms
 6  0.so-5-0-0.TL1.DCA6.ALTER.NET (152.63.0.142)  77.508 ms  77.866 ms  77.399 ms
 7  0.so-4-2-0.CL1.PHL1.ALTER.NET (152.63.37.77)  82.556 ms  82.682 ms  82.059 ms
 8  POS6-0.GW7.PHL1.ALTER.NET (152.63.39.121)  85.052 ms  82.167 ms  79.855 ms
 9  cavtelOC12-gw.customer.alter.net (65.207.92.238)  82.241 ms  82.135 ms  82.714 ms
10  64.83.67.90 (64.83.67.90)  83.943 ms  84.489 ms  83.822 ms
^C


Also, I have tried static routing these blocks from router1 -> router2,
still does not work.

If I static route these blocks from router1->router3 (another colo router
with external connectivity) --- it works fine.

Any clues where I should start looking to find out why these prefixes
are not working properly between router1->router2 ?

This is the strangest thing I have seen in a long time.

thanks
bill