[c-nsp] 6500 TCAM overflows; certain hosts unreachable?

[Martin Moens]

Nate Carlson <> wrote:
> We're having some really odd issues with a pair of 6500's. We know
> that 
> our TCAM table is overflowed, but it's worked fine up until now (new
> pair 
> of SUP720-10GE's on order, but not here yet, of course.)
> 
> Here's the TCAM errors we are getting, which are pretty typical:
> 
> Dec  3 10:29:18: %MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception, Some
> entries will be software switched Dec  3 10:31:49:
> %MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception, Some entries will be
> software switched Dec  3 10:38:10: %MLSCEF-SP-7-FIB_EXCEPTION: FIB
> TCAM exception, Some entries will be software switched  

I had exactly the same situation with a rsp720-3c 


> for a NPE-G2. After that, we started having intermittent connectivity
> issues to various IP's on the internet. When we saw those issues, we
> swapped the G1 back in, with the same config (verified via Rancid.)
> 
> From our hosts connected to the 6500's, some remote IP's work fine,

Sounds very familiar ;-)

 
> Now, the real odd part, is that from the same 6509, coming from the
> .14 
> address, I can hit those IP's without any issues:

Same here
 
> Are these the type of issues expected with TCAM overflows? It seems
> odd to 
> me that our CPU utilization would be low, but we'd be having these,
> unless 'sh proc cpu' isn't the right place to look for that?

Yes.


> Appreciate any thoughts. If we can definitively say that TCAM is the
> issue, we'll filter our BGP routes (get rid of the /24's).. my
> understanding is that to get hardware-switched routes again, though,
> we'd have to reboot the 6500 - is that also correct?
> 
> Thanks much!
> 
> -Nate

I solved my problem by requesting my upstreams to provide me with a default
route, and only have my IX sessions unfiltered. This brought the number of
routes back to ~60K which was ok after a reboot. 
Beginning this week the 3c was replaced with a 3cxl, everything works again
as it should.
Reboot was really needed btw.

Good luck....

Martin