[nsp] eBGP routes not balancing

Jeff Chan cisco-nsp at jeffchan.com
Mon Nov 17 02:50:31 EST 2003


Hi all,
We're running multihomed with two DS3s and have been for a while.
After we reset our peering session with one upstream our outbound
traffic no longer balances correctly.  Both are large providers
and we're getting similarly full routes from both.  Our route maps
are stamping identical MEDs and localprefs on all the incoming
routes but the bestpath selection is clearly skewed towards one
(AT&T which was reset least recently).  I'm wondering if anyone
has any ideas how to correct this. 

A couple possibilities are that the BGP decision algorithm is
reaching near the bottom where peer addresses are considered
and the peer with the lower numerical IP address is chosen,
or there is some kind of historical precedence issue where the
older route is favored.

I'm hoping this is a fairly easy one for you guys, but hope
it's not too obvious.  :)  Here are some configs.  This is
on a 7513 with CX-HIPs for the DS3s.  IOS is 12.2(14)S3.
CEF is currently disabled for other compatibility issues
but presumably is irrelevant for CX boards, and the balancing
was a lot closer to even before with the same hardware, IOS,
and configs....

interface Hssi4/0
 description AT&T DS3
 bandwidth 10000
 ip address 12.118.243.166 255.255.255.252
 serial restart-delay 0
 no cdp enable
 hold-queue 512 in
 hold-queue 512 out

interface Hssi11/0
 description Sprint DS3
 bandwidth 12000
 ip address 144.228.52.182 255.255.255.252
 serial restart-delay 0
 no cdp enable
 hold-queue 512 in
 hold-queue 512 out

router bgp 4150
 no synchronization
 bgp always-compare-med
 bgp log-neighbor-changes
 bgp dampening
 network nnn...
 neighbor 12.118.243.165 remote-as 7018
 neighbor 12.118.243.165 description AT&T
 neighbor 12.118.243.165 version 4
 neighbor 12.118.243.165 prefix-list SUPRANET-NETWORKS out
 neighbor 12.118.243.165 route-map STANDARD-ROUTE in
 neighbor 12.118.243.165 filter-list 11 out
 neighbor 144.228.52.181 remote-as 1239
 neighbor 144.228.52.181 description Sprint
 neighbor 144.228.52.181 version 4
 neighbor 144.228.52.181 prefix-list SUPRANET-NETWORKS out
 neighbor 144.228.52.181 route-map STANDARD-ROUTE in
 neighbor 144.228.52.181 filter-list 11 out
 no auto-summary

route-map STANDARD-ROUTE permit 10
 match ip address 10
 set metric 10
 set local-preference 100

supranet01>sh ip bgp sum
BGP router identifier 66.170.0.1, local AS number 4150
BGP table version is 14615826, main routing table version 14615826
126321 network entries using 12758421 bytes of memory
378215 path entries using 18154320 bytes of memory
142988 BGP path attribute entries using 8579640 bytes of memory
63915 BGP AS-PATH entries using 1673176 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
1 BGP filter-list cache entries using 12 bytes of memory
BGP using 41165593 total bytes of memory
Dampening enabled. 189 history paths, 176 dampened paths
BGP activity 387767/261446 prefixes, 5443140/5064925 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
12.118.243.165  4  7018 2264780   95888 14615812    0    0 1d23h      125820
144.228.52.181  4  1239 5113516  172568 14615826    0    0 1d23h      126054
[...]

supranet01>sh ip bgp 4.0.0.0
BGP routing table entry for 4.0.0.0/8, version 14369789
Paths: (3 available, best #2, table Default-IP-Routing-Table)
  Not advertised to any peer
  1239 3356, (aggregated by 3356 4.68.0.3)
    144.228.52.181 from 144.228.52.181 (144.228.243.27)
      Origin IGP, metric 10, localpref 100, valid, external, atomic-aggregate
  7018 3356, (aggregated by 3356 4.68.0.3)
    12.118.243.165 from 12.118.243.165 (12.123.216.79)
      Origin IGP, metric 10, localpref 100, valid, external, atomic-aggregate, best
  701 3356, (aggregated by 3356 4.68.0.3)
    137.39.7.236 from 137.39.7.236 (137.39.7.236)
      Origin IGP, metric 200, localpref 50, valid, external, atomic-aggregate

(UUNet (AS701) is used as a backup connection.  With a localpref of 50
their routes won't have any effect unless the others peers are down.)

The bestpath stats show Sprint routes being selected much less of
the time (nearly 1 in 3):

AT&T:
    Used as bestpath:             n/a      89915
Sprint:
    Used as bestpath:             n/a      36170


supranet01>sh ip bgp neigh
BGP neighbor is 12.118.243.165,  remote AS 7018, external link
 Description: AT&T
  BGP version 4, remote router ID 12.123.216.79
  BGP state = Established, up for 1d23h
  Last read 00:00:16, hold time is 180, keepalive interval is 60 seconds
  Neighbor capabilities:
    Route refresh: advertised and received(old & new)
    Address family IPv4 Unicast: advertised and received
  Message statistics:
    InQ depth is 0
    OutQ depth is 0
                         Sent       Rcvd
    Opens:                  4          4
    Notifications:          0          0
    Updates:                4    2168915
    Keepalives:         95882      95876
    Route Refresh:          0          0
    Total:              95890    2264795
  Default minimum time between advertisement runs is 30 seconds

 For address family: IPv4 Unicast
  BGP table version 14615847, neighbor version 14615836
  Index 5, Offset 0, Mask 0x20
  Inbound path policy configured
  Outbound path policy configured
  Outgoing update prefix filter list is SUPRANET-NETWORKS
  Outgoing update AS path filter list is 11
  Route map for incoming advertisements is STANDARD-ROUTE
                                 Sent       Rcvd
  Prefix activity:               ----       ----
    Prefixes Current:               3     125820 (Consumes 6042912 bytes)
    Prefixes Total:                 3     229063
    Implicit Withdraw:              0      84939
    Explicit Withdraw:              0      18304
    Used as bestpath:             n/a      89915
    Used as multipath:            n/a          0
    History paths:                n/a         74 (Consumes 3552 bytes)

                                   Outbound    Inbound
  Local Policy Denied Prefixes:    --------    -------
    prefix-list                      205140          0
    AS_PATH loop:                       n/a          1
    Bestpath from this peer:         201224        n/a
    No source path table ID:         205143        n/a
    Total:                           611507          1
  Number of NLRIs in the update sent: max 3, min 0

  Connections established 4; dropped 3
  Last reset 1d23h, due to User reset
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Local host: 12.118.243.166, Local port: 13032
Foreign host: 12.118.243.165, Foreign port: 179

Enqueued packets for retransmit: 0, input: 0  mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x265BC2814):
Timer          Starts    Wakeups            Next
Retrans          2957         94             0x0
TimeWait            0          0             0x0
AckHold         15343      13121             0x0
SendWnd             0          0             0x0
KeepAlive           0          0             0x0
GiveUp              0          0             0x0
PmtuAger            0          0             0x0
DeadWait            0          0             0x0

iss: 1043433219  snduna: 1043487684  sndnxt: 1043487684     sndwnd:  15263
irs: 3775095062  rcvnxt: 3779964682  rcvwnd:      16384  delrcvwnd:      0

SRTT: 300 ms, RTTO: 303 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 736 ms, ACK hold: 200 ms
Flags: higher precedence, nagle

Datagrams (max data segment is 4430 bytes):
Rcvd: 17875 (out of order: 0), with data: 15671, total data bytes: 4869619
Sent: 17053 (retransmit: 94, fastretransmit: 0), with data: 2862, total data byt
es: 54464

BGP neighbor is 144.228.52.181,  remote AS 1239, external link
 Description: Sprint
  BGP version 4, remote router ID 144.228.243.27
  BGP state = Established, up for 1d23h
  Last read 00:00:11, hold time is 135, keepalive interval is 45 seconds
  Neighbor capabilities:
    Route refresh: advertised and received(old & new)
    Address family IPv4 Unicast: advertised and received
  Message statistics:
    InQ depth is 0
    OutQ depth is 0
                         Sent       Rcvd
    Opens:                  3          3
    Notifications:          0          0
    Updates:                3    5113425
    Keepalives:        172565        110
    Route Refresh:          0          0
    Total:             172571    5113538
  Default minimum time between advertisement runs is 30 seconds

 For address family: IPv4 Unicast
  BGP table version 14615850, neighbor version 14615850
  Index 4, Offset 0, Mask 0x10
  Inbound path policy configured
  Outbound path policy configured
  Outgoing update prefix filter list is SUPRANET-NETWORKS
  Outgoing update AS path filter list is 11
  Route map for incoming advertisements is STANDARD-ROUTE
                                 Sent       Rcvd
  Prefix activity:               ----       ----
    Prefixes Current:               3     126056 (Consumes 6052560 bytes)
    Prefixes Total:                 3     243574
    Implicit Withdraw:              0     102838
    Explicit Withdraw:              0      14680
    Used as bestpath:             n/a      36170
    Used as multipath:            n/a          0
    History paths:                n/a         39 (Consumes 1872 bytes)

                                   Outbound    Inbound
  Local Policy Denied Prefixes:    --------    -------
    route-map:                            0          1
    prefix-list                      177193          0
    Bestpath from this peer:          59669        n/a
    No source path table ID:         177196        n/a
    Total:                           414058          1
  Number of NLRIs in the update sent: max 3, min 0

  Connections established 3; dropped 2
  Last reset 1d23h, due to User reset
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Local host: 144.228.52.182, Local port: 13033
Foreign host: 144.228.52.181, Foreign port: 179

Enqueued packets for retransmit: 0, input: 0  mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x265BC2E70):
Timer          Starts    Wakeups            Next
Retrans          4035        236             0x0
TimeWait            0          0             0x0
AckHold         12478      11328             0x0
SendWnd             0          0             0x0
KeepAlive           0          0             0x0
GiveUp              0          0             0x0
PmtuAger            0          0             0x0
DeadWait            0          0             0x0

iss: 3382838677  snduna: 3382910907  sndnxt: 3382910907     sndwnd:  31666
irs: 4244662677  rcvnxt: 4250898248  rcvwnd:      12709  delrcvwnd:   3675

SRTT: 322 ms, RTTO: 478 ms, RTV: 156 ms, KRTT: 0 ms
minRTT: 4 ms, maxRTT: 596 ms, ACK hold: 200 ms
Flags: higher precedence, nagle

Datagrams (max data segment is 4430 bytes):
Rcvd: 16731 (out of order: 0), with data: 13026, total data bytes: 6235570
Sent: 16408 (retransmit: 236, fastretransmit: 0), with data: 3798, total data by
tes: 72229


Any ideas or suggestions would be welcomed with gratitude.

Cheers,

Jeff C.
-- 
Jeff Chan
mailto:cisco-nsp at jeffchan.com
http://www.supranet.net/



More information about the cisco-nsp mailing list