[c-nsp] MPLS MTU / Jumbo frames etc.

[Brandon Applegate]

I know this has been covered, at least in part on this list before, and I 
have read those posts.  However, I'm still trying to wrap my head around 
what is happening internally (or rather on the wire) in the various 
scenarios.

Scenario #1
===========
10 gig interface (ES20 CXL based) - default mtu 1500
MPLS turned on, no 'mpls mtu' command

Default, packets have one label, I get icmp 3,4 frag needed back telling 
me to go to 1496 for 1500 byte (linux ping -M do -s 1472)

Scenario #2
===========
10 gig interface (ES20 CXL based) - default mtu 1500
MPLS turned on - 'mpls mtu override 1508' added

Default, packets have one label, packet is '1504', no icmp frag
Interface can do up to 9216 mtu, so 1500+N labels == not a big deal (??)

Scenario #3
===========
10 gig interface (ES20 CXL based) - mtu changed to 9216
MPLS turned on - mpls mtu == interface mtu by default (does not show up in 
config)

One label packet, with 9216 size (linux ping -M do -s 9188) goes through, 
with no icmp frag needed.

So I'm confused on what's happening in one scenario vs. another.  It seems 
that in scenario 1, the 'outer' MTU is 'signalling' down and kicking off a 
icmp frag needed.

Scenario 2 goes through because we are telling the router it's allowed to 
send a 'baby-giant' (i hate that term).

Scenario 3 really gets me though.  Why doesnt it complain and tell me icmp 
frag to 9212 or something ?  Isnt the frame 9220 when it's all said and 
done ?  Is the router fragmenting this in software at the 'mpls level' and 
just not telling me ?  Should I set mtu down to 9212 or something to make 
sure that the router NEVER frags frames ?

I guess a fireaxe solution would be for us to simply define 'jumbo frames' 
in our network as 9000 bytes, period.  But I'd like to actually understand 
why this behaviour seems to change as I slide the MTU around.  I want to 
make sure that our $$$ isnt being wasted by killing the CPU with 
fragmentation (if thats whats happening, again scenario 3 is really 
puzzling me).

Apologizes ahead of time if all this info is out there somewhere, again 
I've read Ivan's page on this, CCO docs, archives etc.  Thanks in advance.

--
Brandon Applegate - CCIE 10273
PGP Key fingerprint:
7407 DC86 AA7B A57F 62D1 A715 3C63 66A1 181E 6996
"SH1-0151.  This is the serial number, of our orbital gun."