[c-nsp] Cisco CEF Hashing algorithm

Wed Dec 27 15:26:46 EST 2006

I'm having a weird problem where some of my IPs can web browse and  
some cannot.  If I go onto a working IP I can traceroute -P TCP -p 80  
to the destination website just fine.  If I use a broken IP the  
traceroute dies at a border between ATT & Alternet.   It is a very  
strange problem,  I assume is being caused by a CEF per-flow load  
balancing algorithm being used on the ATT Border router and sending  
packets from my broken IP down a broken link.  Does anyone know what  
the hashing algorithm is so I can plug source & dest ip & ports into   
it to figure out which link it is.  ATT can't seem to find the  
problem, I find it hard to believe I'm the only one affected.

I have checked everything on my end,  I don't have any CEF load  
balancing,  I don't have any ACLs on the ports with the execption of  
'ip verify unicast reachable-via rx'

We've spoken to the network guys at the other end  
(www.mydoitbest.com) and they never see the inbound port 80 traffic,   
they do see all other traffic (pings, tcp:443) from the broken IP,  
just not port 80.

Any ideas?

Example traceroutes:

WHEN MY IP IS 161.77.99.203 TRACEROUTE WORKS BUT WEB BROWSING DOES NOT

TRACEROUTE FROM 161.77.99.203 TO WWW.MYDOITBEST.COM WORKS

  matthew$ traceroute www.mydoitbest.com
traceroute to www.mydoitbest.com (216.37.56.70), 64 hops max, 40 byte  
packets
  1  161.77.99.1 (161.77.99.1)  17.049 ms  0.794 ms  0.531 ms
  2  gw-07.spfd.crocker.net (204.97.12.4)  4.621 ms  0.637 ms  0.435 ms
  3  12.125.33.29 (12.125.33.29)  15.353 ms  2.317 ms  2.111 ms
  4  gbr1-p60.cb1ma.ip.att.net (12.123.40.138)  8.056 ms  10.683 ms   
8.003 ms
  5  tbr1-p013401.cb1ma.ip.att.net (12.122.11.193)  15.084 ms  8.551  
ms  8.336 ms
  6  tbr2-cl16.n54ny.ip.att.net (12.122.10.22)  8.774 ms  10.752 ms   
8.733 ms
  7  ggr3-ge140.n54ny.ip.att.net (12.122.80.237)  16.546 ms  7.795  
ms  7.787 ms
  8  0.so-1-0-1.br2.nyc4.alter.net (204.255.168.1)  8.472 ms  12.134  
ms  8.143 ms
  9  0.ge-5-1-0.xl4.nyc4.alter.net (152.63.3.121)  15.497 ms  8.123  
ms  8.165 ms
10  0.so-7-0-0.cl2.ind6.alter.net (152.63.65.6)  32.057 ms  32.226  
ms  32.148 ms
11  190.atm7-0.gw5.ind1.alter.net (152.63.68.245)  34.012 ms  34.090  
ms  34.170 ms
12  onecall-pos-core-gw1.customer.alter.net (63.122.162.214)  36.915  
ms  36.754 ms  36.779 ms
13  room104a-2-cedar.nframe.com (216.37.0.114)  36.932 ms  36.943 ms   
37.200 ms
14  ip-216-37-55-88.nframe.com (216.37.55.88)  38.066 ms  39.084 ms   
38.032 ms
^C

TRACEROUTE FROM 161.77.99.203 TO WWW.MYDOITBEST.COM USING TCP:80 DOES  
NOT WORK

traceroute -P tcp -p 80 www.mydoitbest.com
traceroute to www.mydoitbest.com (216.37.56.70), 64 hops max, 52 byte  
packets
  1  161.77.99.1 (161.77.99.1)  17.184 ms  0.853 ms  0.747 ms
  2  gw-07.spfd.crocker.net (204.97.12.4)  4.675 ms  0.628 ms  0.318 ms
  3  12.125.33.29 (12.125.33.29)  15.261 ms  2.162 ms  2.121 ms
  4  gbr1-p60.cb1ma.ip.att.net (12.123.40.138)  8.171 ms  11.324 ms   
8.083 ms
  5  tbr1-p013401.cb1ma.ip.att.net (12.122.11.193)  48.140 ms  42.920  
ms  8.421 ms
  6  tbr2-cl16.n54ny.ip.att.net (12.122.10.22)  15.163 ms  8.626 ms   
8.585 ms
  7  ggr3-ge140.n54ny.ip.att.net (12.122.80.237)  8.628 ms^C

WHET I SET MY IP TO 161.77.99.202 I CAN WEB BROWSE TO WWW.MYDOITBEST.COM

TRACEROUTE FROM 161.77.99.202 TO WWW.MYDOITBEST.COM USING TCP:80 WORKS

:~ matthew$ traceroute -P tcp -p 80 www.mydoitbest.com
traceroute to www.mydoitbest.com (216.37.56.70), 64 hops max, 52 byte  
packets
  1  161.77.99.1 (161.77.99.1)  7.058 ms  0.543 ms  0.506 ms
  2  gw-07.spfd.crocker.net (204.97.12.4)  10.664 ms  0.623 ms  0.322 ms
  3  12.125.33.29 (12.125.33.29)  15.521 ms  2.552 ms  2.459 ms
  4  gbr1-p60.cb1ma.ip.att.net (12.123.40.138)  8.695 ms  10.674 ms   
8.064 ms
  5  tbr1-p013401.cb1ma.ip.att.net (12.122.11.193)  14.986 ms  9.386  
ms  9.253 ms
  6  tbr2-cl16.n54ny.ip.att.net (12.122.10.22)  11.683 ms  10.920 ms   
8.534 ms
  7  ggr3-ge90.n54ny.ip.att.net (12.123.0.101)  15.175 ms  8.051 ms   
7.842 ms
  8  0.so-1-0-1.br2.nyc4.alter.net (204.255.168.1)  8.706 ms  11.403  
ms  7.991 ms
  9  0.ge-5-1-0.xl3.nyc4.alter.net (152.63.3.125)  15.242 ms  8.049  
ms  8.357 ms
10  0.so-7-0-0.cl1.ind6.alter.net (152.63.64.218)  32.686 ms  32.551  
ms  32.312 ms
11  191.atm6-0.gw5.ind1.alter.net (152.63.68.241)  33.349 ms  33.073  
ms  33.000 ms
12  onecall-pos-core-gw1.customer.alter.net (63.122.162.214)  38.111  
ms  37.389 ms  37.377 ms
13  room104a-2-cedar.nframe.com (216.37.0.114)  37.884 ms  37.453 ms   
37.387 ms
14  ip-216-37-55-88.nframe.com (216.37.55.88)  38.330 ms  38.151 ms   
39.514 ms
^C

THE DIFFERENCE BETWEEN THESE TWO TRACEROUTE POINTS TO A PROBLEM  
BETWEEN HOPS 7 & 8

  7  ggr3-ge90.n54ny.ip.att.net (12.123.0.101)  15.175 ms  8.051 ms   
7.842 ms
  8  0.so-1-0-1.br2.nyc4.alter.net (204.255.168.1)  8.706 ms  11.403  
ms  7.991 ms

--
Matthew S. Crocker
Vice President
Crocker Communications, Inc.
Internet Division
PO BOX 710
Greenfield, MA 01302-0710
http://www.crocker.com