[j-nsp] EX4200 missing ARP entry work-around

[Jeff Wheeler]

We have decided on a better work-around for our missing ARP entry
problems on the EX4200 and friends.

As you may recall, the EX4200 will sometimes have an ARP entry in the
control-plane, but it will not be present in the data-plane.  You can
investigate by checking your destination IP address with the command
`show halp-rt route ip rtt-index 0 prefix 192.0.2.2 prefix-length 32`
which will produce output like this:

PFEM0(vty)# show halp-rt route ip rtt-index 0 prefix 192.0.2.2 prefix-length 32

  Route                    Type Paths RtIdx Rpf SipSa Row:Col:Row:Col
  --------------------     ---- ----- ----- --- ----- ---------------
  192.0.2.2              ECMP 0     2037  No  No    439:0:0:0

  Dev0 (RtIdx:2037)
  -----------------
  Command           : Route              CpuCode        : 3
  AppSpCpuCodeEn    : 0                  UcSipFiltEna   : 0
  TtlDecEna         : 1                  TtlOptChkBypass: 0
  IngressMirror     : 0                  QoSProfileEn   : 0
  QoSProfileIndex   : 0                  QoSPrecedence  : 1
  ModifyUp          : 2                  ModifyDscp     : 0
  CounterSet        : 2                  ArpBc2Cpu      : 0
  SipAccessLevel    : 0                  DipAccessLevel : 0
  IcmpRedirExpnMirr : 0                  MtuProfileIdx  : 2
  Ipv6ScopeCheckEn  : 0                  Ipv6DstSiteId  : 0
  NhTnnl            : 0                  NhTnnlIdx      : 0
  NhVlan            : 6                  NhIf           : VIDX4095
  NhArpIdx          : 138
Device: 0
  ArpEntry Idx 138  : 00:2b:f0:19:87:01
  Hit/Miss          : N

Notice above a reference to NhArpIdx 138.  In order for forwarding to
work correctly, there must be an entry # 138 in the `halp-nh
arp-table.`  Since there isn't one, the NhIf shown is VIDX4095, not a
port.  However, if you want to verify that there is no NhArpIdx 138 in
the hardware, you can examine the table with `show halp-nh arp-table`
and scroll down to where # 138 should be.  You won't find it!

PFEM0(vty)# show halp-nh arp-table
Device: 0
... lots of scrolling ...
  ArpEntry Idx 136  : 00:18:8b:f8:b6:6e
  ArpEntry Idx 137  : 00:0e:b6:2d:01:a0
  ArpEntry Idx 139  : 00:19:b9:f9:24:2a
  ArpEntry Idx 140  : 78:2b:cb:3c:91:60

How do you get the switch to populate that entry?  Well, since `clear
arp` on the EX4200 doesn't actually do what it is supposed to do, what
we have been doing in the past is deleting whole subnets, impacting
potentially many machines, and then re-adding them.  This is not good
but it does work.

Our new solution is much better.  We just add a bogus static arp entry
for 192.0.2.2, pointing to some made-up MAC address, and then we
commit, roll back, and commit again.  Like magic, the ARP entry will
re-populate correctly.  Almost as if you really did have a `clear arp`
command on the EX4200, one that worked right!

After adding and then deleting the bogus static ARP for 192.0.2.2 you
can re-examine the PFE and see the results.  Also, your customer's
Internets will begin working correctly again.

I hope this helps.
-- 
Jeff S Wheeler <jsw at inconcepts.biz>
Sr Network Operator  /  Innovative Network Concepts