[c-nsp] Strange MPLS ICMP (mtu) issue

[Neil Robst]

Hi All,

I've been banging my head for several days against a strange issue with
MPLS over GRE over IPSEC and wonder if anyone can help to shed any light
on this please!

The scenario is as follows:

Hardware
Rtr1: Cisco 7206VXR (NPE-G1) running c7200-spservicesk9-mz.151-4.M2.bin
Rtr2: Cisco 2801 running c2801-spservicesk9-mz.151-4.M4.bin

Setup:
Firewalls in front of each router host an IPSEC VPN tunnel which secure's
a GRE tunnel between the two routers (loopback(Lo150) on each router to be
precise)
Each router has mpls ip configured in the tunnel config
Each router runs OSPF redistributing the tunnel point2point subnet and
loopback100 addresses (OSPF router-id) in area 0 (into default VRF)
Each router runs MP-eBGP peering with the other router's looback100
interface and configured for two VRFs - VRF_A and VRF_B
Each router has a loopback interface in VRF_B (for management) and an
ethernet interface in VRF_A (corp WAN)

Problem:
If I attempt to send an ICMP ping across the GRE tunnel (from the 7206 to
the 2801) with a packet size of 1445 bytes and the DF bit set to an IP (in
either VRF_A OR VRF_B) that's on the 2800 (at the other end of the GRE
tunnel) I correctly get an ICMP type 3 code 4 reply back as expected:

The 7206 shows this (with debug ip icmp)
MPLS: ICMP: dst (10.103.2.12) frag. needed and DF set unreachable sent to
172.18.4.7

And in my terminal I get:
$ ping -M do -s 1445 10.103.2.12 -c 5
PING 10.103.2.12 (10.103.2.12) 1445(1473) bytes of data.
>From 172.18.4.7 icmp_seq=1 Frag needed and DF set (mtu = 1472)
>From 172.18.4.7 icmp_seq=1 Frag needed and DF set (mtu = 1472)
>From 172.18.4.7 icmp_seq=1 Frag needed and DF set (mtu = 1472)
>From 172.18.4.7 icmp_seq=1 Frag needed and DF set (mtu = 1472)
>From 172.18.4.7 icmp_seq=1 Frag needed and DF set (mtu = 1472)

--- 10.103.2.12 ping statistics ---
0 packets transmitted, 0 received, +5 errors

That's all fine! However if I add a static route on the 2800 into VRF_A
for a network behind the next hop from VRF_A at that site:


ip route vrf VRF_A 10.103.254.0 255.255.255.0 10.103.2.1

And then attempt to ping an IP in THAT subnet - 10.103.254.10 - with the
DF bit set and a packet size of 1445 I *DON'T* get an ICMP type 3 code 4
reply back - I get nothing (in my terminal). HOWEVER the 7206VXR shows the
expect debug message as before:


MPLS: ICMP: dst (10.103.254.10) frag. needed and DF set unreachable sent
to 172.18.4.7

And after much head scratching I've worked out that the ICMP replies ARE
being sent from the router - but oddly being send UP the GRE tunnel back
to the 2801 and onto it's next hop (in VRF_A) - 10.103.2.1. This is a
Cisco ASA and I can see the error in it's logs:

  
  
  
  
  No matching connection for ICMP error message: icmp src
int-wan-dmz:172.31.248.142 dst outside:172.18.4.7 (type 3, code 4) on
int-wan-dmz interface.  Original IP payload: icmp src 172.18.4.7 dst
10.103.254.10 (type 8, code 0).


172.31.248.142 is the IP of an MPLS-enabled interface on the 7206VXR and
is where I *would* expect these ICMP replies to be sourced from in my
network topology. However the 172.18.4.7 is across this interface
(Gi0/2.1066 on the 7206) and NOT up the GRE tunnel so I'm at a loss as to
why the 7206 is sending these ICMP replies this wayŠ? Especially when it
correctly sends ICMP replies to IP addresses locally on the 2801 itself
back the right way?!

I hope this makes some sort of sense and that someone has come across this
issue before - and has a fix for it!).

Any assistance / pointers very gratefully received!

Best Regards,
Neil