[c-nsp] ARP strangeness

Wed Jan 5 12:46:14 EST 2011

This is a problem with how the linksys sends out the DHCP request for
renewal.  The BEFSR41 and BEFSR81 use a broadcast bootp flag, whereas all
other CPE devices use a unicast bootp flag.  When using the broadcast bootp
flag the Cisco will send an arp broadcast which is dropped by the access
equipment.

On Wed, Jan 5, 2011 at 11:32 AM, Frank Bulk <frnkblk at iname.com> wrote:

> Yes, broken spoke would be one thing to call it.  Another cisco-nsp reader
> guessed what FTTH platform I have, because they've seen the same issue.
> I've also posted on the vendor's closed web forum and I'll see if I get any
> responses there.
>
> With the current settings our test CPE remains a "live spoke".  If it would
> fail, what you're suggesting regarding a span egress would be our next
> step.
>
> Frank
>
> -----Original Message-----
> From: Rodney Dunn [mailto:rodunn at cisco.com]
> Sent: Wednesday, January 05, 2011 7:35 AM
> To: frnkblk at iname.com
> Cc: 'Keegan Holley'; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] ARP strangeness
>
> <snip>
>
> > Broadcast ARPs, due to the FTTH's L2 security protection mechanisms, are
> not
> > passed on.  Once the MAC entry ages out of the FTTH, it has to relearn it
> > from the CPE (I think via ARP) before it will pass thru certain types of
> > traffic.
>
> Whoaa....ok. That means you could have a broken spoke if the main
> traffic is out towards the CPE and no CPE originated traffic. Unlikely
> I'm sure but not impossible.
>
> > The rest of what you're describing about ARP expiration makes sense.
>
> Ok. There were some arp code optimizations that I'd have to dig back up
> and see if it changed any of the legacy behavior but I don't think it did.
>
> What is really needed here would be the span egress the 7600 port
> watching for those unicast refreshes. If that doesn't happen it's a bug
> on the 7600. If they do then it's likely an issue downstream.
>
> Rodney
>
>
> > Thanks,
> >
> > Frank
> >
> > -----Original Message-----
> > From: Rodney Dunn [mailto:rodunn at cisco.com]
> > Sent: Tuesday, January 04, 2011 8:01 PM
> > To: frnkblk at iname.com
> > Cc: 'Keegan Holley'; cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] ARP strangeness
> >
> > On 1/3/11 11:13 PM, Frank Bulk - iName.com wrote:
> >> The 7609 does stop ARPing after receiving a reply from the CPE, but the
> > 7609
> >> ARPs again 7 minutes later.  One person told me off-list that Cisco
> > doesn't
> >> expire an ARP entry before checking its ARP entries by doing an ARP
> > request.
> >> Since ARP timeout is set for 8 minutes, perhaps Cisco's approach is to
> ARP
> >> the host one minute before expiration.
> >
> > There were some changes to ARP at one point to provide some more
> > triggered capability. I don't recall exactly what that was but the
> > default behavior for many years was that we send a unicast arp to the
> > destination 60 seconds prior to the arp timer set to expire. If we don't
> > get a response we send it again when the timer pops and if no response
> > we invalidate the ARP entry.
> >
> >>
> >> Because the FTTH product has its own smart-L2 implementation with a MAC
> >> address expiration time of 7 minutes, it won't forward directed or
> > broadcast
> >> ARP requests from the 7609 once the CPE's MAC address has expired from
> the
> >> FTTH's MAC table.
> >
> > I wonder if you are missing one and then getting in to a race condition
> > of the FTTH product not forwarding. If so it would seem you need to set
> > the arp refresh down to a value less than the timeout of the transport
> gear.
> >
> >
> >     Once the CPE goes deaf, even a full DHCP exchange doesn't
> >> "wake up" the connection.  Only power-cycling the BEFRS41 resolves the
> >> issue.  The difference between a power cycle and a full DHCP exchange is
> >> that the BEFSR41 does an ARP request for the default router (7609) after
> > the
> >> DHCP exchange.
> >
> > If you lose the arp from the 7609 and you ping from it directly we
> > should send a broadcast arp.
> >
> > Not sure where you are getting the packet capture but a span directly
> > off the port is what you need to see if we don't send those two
> > refreshes (one 60 seconds prior to the timer expiring and the second on
> > the timer expiring). If we do send those and no response then data
> > driven from the 7609 (packets headed towards that remote IP) should
> > trigger the arp broadcast out again.
> >
> > Rodney
> >
> >
> >>
> >> I've extended the MAC address expiration time of the FTTH link to 15
> > minutes
> >> (two times the 7 minutes plus 1 minute) and we'll see how that goes.
> >>
> >> Frank
> >>
> >> -----Original Message-----
> >> From: Keegan Holley [mailto:keegan.holley at sungard.com]
> >> Sent: Monday, January 03, 2011 7:14 PM
> >> To:<frnkblk at iname.com>
> >> Cc:<cisco-nsp at puck.nether.net>
> >> Subject: Re: [c-nsp] ARP strangeness
> >>
> >> The 7600 should stop arping if it has gotten a reply.  What happens when
> > you
> >> ping your test CPE both from the router and from another node behind it?
> >> You can also run he command sh ip cef exact-match to see what the router
> > is
> >> doing with a specific source-destination ip pair. If nothing strange
> pops
> > up
> >> then it's probably a bug.
> >>
> >> Sent from my iPhone
> >>
> >> On Jan 3, 2011, at 12:58 PM, Frank Bulk<frnkblk at iname.com>   wrote:
> >>
> >>> We have over a thousand FTTH customers hanging off a VLAN on our 7609-S
> >>> running 12.2SRE3.  Those who have Linksys BEFRS41 (wired-only routers)
> > are
> >>> complaining about lack of Internet access after many hours or days of
> > idle
> >>> time (not using their PC or other devices).  Those who have Linksys
> > WRT54G
> >>> (wireless) have no complaints (my guess is that they're sending packets
> >> out
> >>> regularly).
> >>>
> >>> We replicated this in our CO and put a hub between the ONT and the
> > Linksys
> >>> CPE so that we could capture those packets.  What we're seeing in that
> >>> capture are directed ARP requests every 7 minutes from the 7609 to the
> >>> Linksys with an ARP response from the Linksys.  After many hours, the
> >> 7609-S
> >>> stops sending the ARP requests (well, at least we're not seeing it come
> >> in,
> >>> perhaps it did try).
> >>>
> >>> We currently have our ARP timeout set to 480 seconds and MAC address
> > table
> >>> aging time to 540 seconds.  Why?  We use "mac-address-table
> synchronize"
> >>> which is set to 160 seconds by default.  The recommendation from that
> >>> command is to set ARP three times that, so that would be 480.  But it's
> >> also
> >>> recommended that the MAC address table aging time be greater than the
> ARP
> >>> timeout, so we added another 60 seconds on top.
> >>>
> >>> Two questions:
> >>> - why is the 7609 sending any directed ARP requests at all, every 7
> >> minutes?
> >>> - why does it appear to stop sending them after many hours?
> >>>
> >>> I'm all ears if we should be using different expiration values, but the
> >>> numbers I'm using are based on reading a lot of cisco-nsp archives and
> >> Cisco
> >>> tech articles.
> >>>
> >>> Frank
> >>>
> >>> _______________________________________________
> >>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> >>> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >>>
> >>
> >>
> >> _______________________________________________
> >> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>

-- 
Chad Whitten
chadwick.whitten at gmail.com
601-519-4172