[nsp] BGP issues after upgrading IOS versions

Mon Aug 25 18:58:42 EDT 2003

We've been having some router stability issues ever since these worms
showed up.  We saw 100% CPU utilization until we added the ICMP filters.
We learned how to use Netflow data to identify compromised hosts.  Yea!
That is the one good thing in all this.  The ICMP filters have turned 5
to 10 minute outages into 20 second outages.

However, with all our filters in place, we still have issues with
our border router, a 7206VXR with NPE300.  It handles BGP to our two
upstreams and some T1 traffic on CT3 cards.  We get full views from both
upstreams and advertise our one /19 block.  It is also still running a
non-fixed IOS from before the cisco-sa-20030717-blocked advisory.  We
installed the filters and waited for the dust to settle since I was
going out of town the next day.

The border router drops OSPF connections and loses carrier on all T1 and
Ethernet interfaces and stops responding on the serial console from time
to time for between 10 and 20 seconds at a time.

In hopes of curing this we tried to upgrade the IOS over the weekend.
We have tried 12.0(25)S1 and 12.2(18)S.  In both cases, everything
looked like it worked with one exception.  We could not reach anywhere
outside our network.

With 12.0(25)S1, we received the full BGP routes from both providers,
but simply couldn't go anywhere.  One provider told us they were not
seeing any advertisements from us.  On reverting to the old IOS, we
suffered the same symptoms until we copied the original config off tftp
into the running config.  Ten seconds or less after that, we had full
external connectivity.  The diff between the non-working and working
configs showed no access-list, route-map, or router BGP changes.

With 12.2(18)S, the BGP sessions kept dieing with "out of memory"
errors.  The router has 155MB according to "sho ver".

cisco 7206VXR (NPE300) processor with 155648K/40960K bytes of memory.
R7000 CPU at 262Mhz, Implementation 39, Rev 2.1, 256KB L2, 2048KB L3 Cache
6 slot VXR midplane, Version 2.0

I know I've got to be doing something stupid or overlooking the obvious
somewhere.  Our BGP-foo is very weak here.  

Has anyone heard of not being able to advertise your netblocks after an
IOS change?

router bgp 4276
 no synchronization
 bgp log-neighbor-changes
 network 216.223.192.0 mask 255.255.224.0
 neighbor 209.123.10.33 remote-as 8001
 neighbor 209.123.10.33 description NAC.net
 neighbor 209.123.10.33 update-source FastEthernet0/0
 neighbor 209.123.10.33 version 4
 neighbor 209.123.10.33 distribute-list 40 out
 neighbor 209.123.10.33 route-map 8001-prepend in
 neighbor 209.123.10.33 route-map 4276-prepend out
 neighbor 216.213.101.237 remote-as 12124
 neighbor 216.213.101.237 description Thorn Communications
 neighbor 216.213.101.237 update-source FastEthernet6/0.3
 neighbor 216.213.101.237 version 4
 neighbor 216.213.101.237 distribute-list 40 out
 no auto-summary
!
access-list 40 permit 216.223.192.0 0.0.31.255
!
route-map 4276-prepend permit 10
 set as-path prepend 4276 4276 4276 4276 4276
!
route-map 8001-prepend permit 10
 set as-path prepend 8001 8001

Extended IP access list 115
    permit tcp host 209.123.10.33 host 209.123.10.34 eq bgp
    permit tcp host 209.123.10.33 eq bgp host 209.123.10.34
    permit tcp host 216.213.101.237 host 216.213.101.238 eq bgp
    permit tcp host 216.213.101.237 eq bgp host 216.213.101.238
    permit ip 216.223.192.0 0.0.31.255 any (83487688 matches)
    deny ip any any log-input (122 matches)
Extended IP access list 116
    deny ip host 0.0.0.0 any (122 matches)
    deny ip 127.0.0.0 0.255.255.255 any (10 matches)
    deny ip 192.0.2.0 0.0.0.255 any
    deny ip 224.0.0.0 0.15.255.255 any
    deny ip 10.0.0.0 0.255.255.255 any (6218 matches)
    deny ip 172.16.0.0 0.0.15.255 any (119 matches)
    deny ip 192.168.0.0 0.0.255.255 any (1565 matches)
    permit tcp host 209.123.10.33 host 209.123.10.34 eq bgp
    permit tcp host 209.123.10.33 eq bgp host 209.123.10.34 (12218 matches)
    permit tcp host 216.213.101.237 host 216.213.101.238 eq bgp (29310 matches)
    permit tcp host 216.213.101.237 eq bgp host 216.213.101.238
    deny ip 216.223.192.0 0.0.31.255 any log
    deny ip any host 209.123.10.34 (1032 matches)
    deny ip any host 216.213.101.238 (110890 matches)
! snipped a lot of denys to router interface IPs
    permit ip any any (76097848 matches)

Diff from working to config to non-working 12.0(25)S1 running-config:

--- 120-20.3-S1        Mon Aug 25 03:22:44 2003
+++ 120-25-S1      Mon Aug 25 04:11:54 2003
! boot system flash lines changed.
@@ -43,6 +42,8 @@
 ip name-server 216.223.198.28
 ip ssh time-out 120
 ip ssh authentication-retries 3
+mpls ldp logging neighbor-changes
+no mpls traffic-eng auto-bw timers frequency 0
 !
 !
 controller T3 5/0
@@ -769,7 +773,7 @@
  no auto-summary
 !
 ip classless
-ip route 0.0.0.0 0.0.0.0 209.123.10.33
+ip route 0.0.0.0 0.0.0.0 209.123.10.33 250
 ip route 0.0.0.0 0.0.0.0 216.213.101.237
 ip route 1.0.0.0 255.0.0.0 Null0
 ip route 2.0.0.0 255.0.0.0 Null0
@@ -861,7 +865,6 @@
 ip route 207.193.136.1 255.255.255.255 Null0
 ip route 211.139.36.162 255.255.255.255 Null0
 ip route 211.154.128.160 255.255.255.240 Null0
-ip route 216.223.192.0 255.255.224.0 Null0 240
 ip route 216.223.192.0 255.255.224.0 Null0 255
 ip route 216.223.192.53 255.255.255.255 FastEthernet6/0
 ip route 216.223.193.0 255.255.255.0 216.223.198.40
!
! plus differences in snmp-server directive statements.


-- 
Scott Lambert           KC5MLE           System Administrator

Attention Customers:
  Refer-A-Friend and receive one month of service for free!
  For further details, please visit: http://inch.com/refer-a-friend/
  or, call the Internet Channel billing department at (212) 243-5200.
