[nsp] BGP issues after upgrading IOS versions

Scott Lambert scott at inch.com
Mon Aug 25 18:58:42 EDT 2003

We've been having some router stability issues ever since these worms
showed up.  We saw 100% CPU utilization until we added the ICMP filters.
We learned how to use Netflow data to identify compromised hosts.  Yea!
That is the one good thing in all this.  The ICMP filters have turned 5
to 10 minute outages into 20 second outages.

However, with all our filters in place, we still have issues with
our border router, a 7206VXR with NPE300.  It handles BGP to our two
upstreams and some T1 traffic on CT3 cards.  We get full views from both
upstreams and advertise our one /19 block.  It is also still running a
non-fixed IOS from before the cisco-sa-20030717-blocked advisory.  We
installed the filters and waited for the dust to settle since I was
going out of town the next day.

The border router drops OSPF connections and loses carrier on all T1 and
Ethernet interfaces and stops responding on the serial console from time
to time for between 10 and 20 seconds at a time.

In hopes of curing this we tried to upgrade the IOS over the weekend.
We have tried 12.0(25)S1 and 12.2(18)S.  In both cases, everything
looked like it worked with one exception.  We could not reach anywhere
outside our network.

With 12.0(25)S1, we received the full BGP routes from both providers,
but simply couldn't go anywhere.  One provider told us they were not
seeing any advertisements from us.  On reverting to the old IOS, we
suffered the same symptoms until we copied the original config off tftp
into the running config.  Ten seconds or less after that, we had full
external connectivity.  The diff between the non-working and working
configs showed no access-list, route-map, or router BGP changes.

With 12.2(18)S, the BGP sessions kept dieing with "out of memory"
errors.  The router has 155MB according to "sho ver".

cisco 7206VXR (NPE300) processor with 155648K/40960K bytes of memory.
R7000 CPU at 262Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 Cache
6 slot VXR midplane, Version 2.0

I know I've got to be doing something stupid or overlooking the obvious
somewhere.  Our BGP-foo is very weak here.  

Has anyone heard of not being able to advertise your netblocks after an
IOS change?

router bgp 4276
 no synchronization
 bgp log-neighbor-changes
 network mask
 neighbor remote-as 8001
 neighbor description NAC.net
 neighbor update-source FastEthernet0/0
 neighbor version 4
 neighbor distribute-list 40 out
 neighbor route-map 8001-prepend in
 neighbor route-map 4276-prepend out
 neighbor remote-as 12124
 neighbor description Thorn Communications
 neighbor update-source FastEthernet6/0.3
 neighbor version 4
 neighbor distribute-list 40 out
 no auto-summary
access-list 40 permit
route-map 4276-prepend permit 10
 set as-path prepend 4276 4276 4276 4276 4276
route-map 8001-prepend permit 10
 set as-path prepend 8001 8001

Extended IP access list 115
    permit tcp host host eq bgp
    permit tcp host eq bgp host
    permit tcp host host eq bgp
    permit tcp host eq bgp host
    permit ip any (83487688 matches)
    deny ip any any log-input (122 matches)
Extended IP access list 116
    deny ip host any (122 matches)
    deny ip any (10 matches)
    deny ip any
    deny ip any
    deny ip any (6218 matches)
    deny ip any (119 matches)
    deny ip any (1565 matches)
    permit tcp host host eq bgp
    permit tcp host eq bgp host (12218 matches)
    permit tcp host host eq bgp (29310 matches)
    permit tcp host eq bgp host
    deny ip any log
    deny ip any host (1032 matches)
    deny ip any host (110890 matches)
! snipped a lot of denys to router interface IPs
    permit ip any any (76097848 matches)

Diff from working to config to non-working 12.0(25)S1 running-config:
--- 120-20.3-S1        Mon Aug 25 03:22:44 2003
+++ 120-25-S1      Mon Aug 25 04:11:54 2003
! boot system flash lines changed.
@@ -43,6 +42,8 @@
 ip name-server
 ip ssh time-out 120
 ip ssh authentication-retries 3
+mpls ldp logging neighbor-changes
+no mpls traffic-eng auto-bw timers frequency 0
 controller T3 5/0
@@ -769,7 +773,7 @@
  no auto-summary
 ip classless
-ip route
+ip route 250
 ip route
 ip route Null0
 ip route Null0
@@ -861,7 +865,6 @@
 ip route Null0
 ip route Null0
 ip route Null0
-ip route Null0 240
 ip route Null0 255
 ip route FastEthernet6/0
 ip route
! plus differences in snmp-server directive statements.

Scott Lambert           KC5MLE           System Administrator

Attention Customers:
  Refer-A-Friend and receive one month of service for free!
  For further details, please visit: http://inch.com/refer-a-friend/
  or, call the Internet Channel billing department at (212) 243-5200.

More information about the cisco-nsp mailing list