[c-nsp] 7600 HFIB bug?

Mon Aug 1 09:51:38 EDT 2011

I am using  12.2(33r)SRB4, TCAM is at 70% for IPv4.

All show commands (sh cef, sh mls, sh mpls, sh ip route, etc) are pointing
to the right places, but still, if I remove the ACL traffic stops for those
destinations.

This is the time when not having a TAC becomes more expensive than having
one...

On Sun, Jul 31, 2011 at 5:00 PM, Kevin Graham <
kgraham at industrial-marshmallow.com> wrote:

> The log ACE's force bypass hardware forwarding and CEF altogether, so your
> speculation on FIB programming is right on. (...and the failure mode matches
> as well).
>
> What does "sh plat hard cap" show? Over-capacity TCAM FIB supposedly got
> much better circa 12.2(33), but is still squirrely.
>
> Additionally, check "sh mls cef ip" for the prefixes in question globally
> and each of the dCEF-enabled modules.
>
> [sent from my mobile]
>
> On Jul 28, 2011, at 12:53 PM, Persio Pucci <persio at gmail.com> wrote:
>
> > Matthew,
> >
> > clear arp, clear ip route, clear cef, nothing helps, I have even reloaded
> SP
> > and Rio routers during a window, tired of this, and still it won't work.
> >
> > I am running 12.2(33r)SRB4 on a RSP720-3CXL-GE. Interfaces have MPLS
> > running. And Multicast, but so they did before when it worked.
> >
> > Before it broke, MPLS was on both to_SP and to_NY interfaces, and I had a
> > MPLS-TE tunnel from SP to NY. When it broke, the tunnel would not work
> and I
> > had to remove it, remove MPLS from the to_NY interface, and make Rio a
> BGP
> > hop for both SP and NY to resume communications.
> >
> > The weirdest part is that when we first brought up the 7600, it was
> working
> > OK. But then we had a hit on our Rio/SP circuit, and when it cam back, it
> > never worked again.
> >
> > This is the to_SP interface
> >
> > interface POS4/1/0
> >> description * TO SPO * ACTIVE
> >> ip address X.X.X.X 255.255.255.252
> >> ip nat outside
> >> ip router isis
> >> ip pim sparse-dense-mode
> >> mpls traffic-eng tunnels
> >> mpls ldp discovery transport-address X.X.X.X
> >> mpls label protocol ldp
> >> mpls ip
> >> crc 32
> >> pos framing sdh
> >> pos scramble-atm
> >> aps group 20
> >> aps working 1
> >> hold-queue 4096 in
> >> hold-queue 4096 out
> >> ip rsvp bandwidth 100000 100000
> >> end
> >
> >
> > This is to NY
> >
> > interface GigabitEthernet1/2
> >> description * TO NY*1 *
> >> ip address X.X.X.X 255.255.255.252 secondary
> >> ip address X.X.X.X 255.255.255.240
> >> ip access-group 123 out
> >> no ip redirects
> >> ip router isis
> >> load-interval 30
> >> mpls mtu 1524
> >> mpls traffic-eng tunnels
> >> mpls ldp discovery transport-address X.X.X.X
> >> mpls label protocol ldp
> >> mpls ip
> >> spanning-tree link-type point-to-point
> >> hold-queue 4096 in
> >> ip rsvp bandwidth 100000 100000
> >> end
> >
> >
> > ACL 123 is the one I have in place in the meanwhile punting the packets I
> > really need to go through:
> >
> > access-list 123 permit ip any X.X.XX 0.0.0.255 log
> >> access-list 123 permit ip X.X.X.X 0.0.0.255 any log
> >> access-list 123 permit ip any host X.X.X.X log
> >> access-list 123 permit ip host X.X.X.X any log
> >> access-list 123 permit ip any X.X.X.X 0.0.0.3 log
> >> access-list 123 permit ip X.X.X.X 0.0.0.3 any log
> >> access-list 123 permit ip any any
> >
> >
> >
> > On Thu, Jul 28, 2011 at 4:37 PM, Matthew Huff <mhuff at ox.com> wrote:
> >
> >> It's very possible the fib is correct, but should be correctable by
> doing a
> >> "clear arp" and a "clear ip route *". What IOS are you running and what
> sup
> >> engine do you have? Also, what does "show ip cef exact-route source_ip
> >> dest_ip" show?
> >>
> >> Are there anything else "interesting" configured? MPLS, PBR, i.e., what
> >> does the interface config look like?
> >>
> >> ----
> >> Matthew Huff             | 1 Manhattanville Rd
> >> Director of Operations   | Purchase, NY 10577
> >> OTA Management LLC       | Phone: 914-460-4039
> >> aim: matthewbhuff        | Fax:   914-460-4139
> >>
> >>
> >> -----Original Message-----
> >> From: cisco-nsp-bounces at puck.nether.net [mailto:
> >> cisco-nsp-bounces at puck.nether.net] On Behalf Of Persio Pucci
> >> Sent: Thursday, July 28, 2011 3:23 PM
> >> To: cisco-nsp at puck.nether.net
> >> Subject: [c-nsp] 7600 HFIB bug?
> >>
> >> Hi all. I am new to the list and this is my first post. :)
> >>
> >> Trying to get to the bottom of a situation, sans-TAC. Long story short,
> for
> >> context sake, I had a 7300 that was replaced by a 7600 at my Rio de
> Janeiro
> >> site connecting to SP and NY.
> >>
> >> (SP --- RIO --- NY)
> >>
> >> Everything was working fine by the time we were finishing replacing the
> >> box,
> >> when our circuit to Sao Paulo was hit and stayed down for about 6 hours.
> >> When the circuit came back up, some communication to NY was just simply
> not
> >> working, the SP rotuer could not reach, for whatever reason, IP
> addresses
> >> that were reachable after replacing the box, before the hit. It used to
> >> work
> >> on a TE tunnel I had to remove and make Rio a BGP hop to put it to work
> >> while I tried to figure wtf was going on. Ever since, I can ping NY's IP
> >> address from Rio, but cannot from SP, altough all routing is in place
> >> (ISIS), all CEF entries are there.
> >>
> >> Well, after a few weeks working on this when time was allowed, I came to
> a
> >> intriguing situation today, while working with the help of a friend. I
> was
> >> trying to debug this by using a permit ACL with log-input on the Rio
> >> interfaces and see what was going on. When I applied the ACL on the
> >> interfaces (ip permit x x log-input, ip permit any any), things started
> >> working, and I was again able to ping from SP to NY. If I remove the
> ACL, I
> >> cease to ping NY from SP.
> >>
> >> I seems like something is borked at the 7600, cause the packets won't go
> >> through if they are CEF switched, but they will when they are punted to
> the
> >> CPU for the logging. Lookis like some FIB/HFIB issue that is beyond
> >> my comprehension.
> >>
> >> Any ideas besides going to TAC? Tks!
> >> _______________________________________________
> >> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >>
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>