[j-nsp] Juniper Ethernet MTU question
Harry Reynolds
harry at juniper.net
Wed Oct 4 15:45:41 EDT 2006
Adding a correction based on some actual testing.
It seems like when mpls mtu is automatically calculated that it subtract
12 bytes (for 3 level label stack) from device mtu. Testing shows that
when you manually configure the mpls family mtu there is no such math,
so you are actually configuring the mpls *pdu*, which includes data and
headers.
With inet and mpls set to 1500 on a device mtu of 1526, I was only able
to generate 1492 byte datagrams as an ingress node. This was 1492, and
not the expected 1496 (single label is needed in this setup) because IP
frag is based on 8 bytes offsets. When I set inet mtu to 1500 and mpls
mtu to 1504 I was able to send full length 1500 bytes packets. If two
ore more labels were needed I would have been back to fragmenting, so
I'm updating recommended jni setting for ios compatibility to the
following:
[edit interfaces ge-0/3/0]
harry at milhouse# show
mtu 1526;
unit 0 {
family inet {
mtu 1500;
address 1.23.1.2/24;
}
family iso;
family mpls {
mtu 1512; <<<<<< allows up to 3 labels while still doing 1500
byte IP.
}
}
[edit interfaces ge-0/3/0]
harry at milhouse# run show interfaces ge-0/3/0
Physical interface: ge-0/3/0, Enabled, Physical link is Up
Interface index: 133, SNMP ifIndex: 27
Link-level type: Ethernet, MTU: 1526, Speed: 1000mbps, Loopback:
Disabled, Source filtering: Disabled,
Flow control: Enabled
Device flags : Present Running
Interface flags: SNMP-Traps 16384
Link flags : None
CoS queues : 4 supported
Current address: 00:90:69:6f:64:5d, Hardware address:
00:90:69:6f:64:5d
Last flapped : 2006-10-04 10:09:44 PDT (02:34:29 ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
Logical interface ge-0/3/0.0 (Index 68) (SNMP ifIndex 31)
Flags: SNMP-Traps Encapsulation: ENET2
Protocol inet, MTU: 1500
Flags: User-MTU
Addresses, Flags: Is-Preferred Is-Primary
Destination: 1.23.1/24, Local: 1.23.1.2, Broadcast: 1.23.1.255
Protocol iso, MTU: 1509
Flags: Is-Primary
Protocol mpls, MTU: 1512
Flags: Is-Primary, User-MTU
> -----Original Message-----
> From: Harry Reynolds
> Sent: Monday, October 02, 2006 8:09 AM
> To: Ihsan Junaidi Ibrahim
> Cc: Juniper-NSP
> Subject: RE: [j-nsp] Juniper Ethernet MTU question
>
> hehe, Now you go and open a can of worms. ;)
>
> Transit LSRs are incapable of performing fragmentation;
> therefore there is potential for resulting discard anytime
> mtus are mismatched.
>
> Our default ethernet mtu settings can lead to fragmentation,
> as you infer, when we are acting as an ingress node for
> transit or locally generated traffic. Here is proof of this behavior:
>
> [edit protocols mpls]
> harry at frank# run ping 10.3.4.1 size 1460 count 1
> PING 10.3.4.1 (10.3.4.1): 1460 data bytes
> 1468 bytes from 10.3.4.1: icmp_seq=0 ttl=64 time=1.932 ms
>
> --- 10.3.4.1 ping statistics ---
> 1 packets transmitted, 1 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 1.932/1.932/1.932/0.000 ms
>
>
> 11:02:05.828893 Out MPLS (label 100000, exp 0, [S], ttl 64),
> IP, length: 1492
>
>
> [edit protocols mpls]
> harry at frank# run ping 10.3.4.1 size 1461 count 1
> PING 10.3.4.1 (10.3.4.1): 1461 data bytes
> 1469 bytes from 10.3.4.1: icmp_seq=0 ttl=64 time=20.195 ms
>
> --- 10.3.4.1 ping statistics ---
> 1 packets transmitted, 1 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 20.195/20.195/20.195/0.000 ms
>
>
> 11:02:10.323250 Out MPLS (label 100000, exp 0, [S], ttl 63),
> IP, length: 1488 11:02:10.323260 Out MPLS (label 100000, exp
> 0, [S], ttl 63), IP, length: 29
>
>
> As I understand this behavior differs from cisco defaults,
> which IIRC would have generated a jumbo in this case (ios
> adds up to 3 labels on top of device mtu, rather than
> reducing payload size to accommodate labels).
>
> Two options present themselves. If you do not want jumbos (or
> fragmentation) set inet mtu to equal mpls mtu (network wide)
> and take efficiency hit associated with smaller packets. Or
> if you can handle jumbos raise media mtu by 12 while forcing
> inet to stay at 1500 (it will automatically adjust when
> device mtu is changed and interface is bounced). Either way
> you end up with an inet *PDU* (which includes IP and TCP
> headers), that matches the underlying mpls *MTU* (which *does
> not* include the mpls headers), and there is no need for
> ingress fragmentation or potential for mtu related discards
> at transit nodes.
>
> The latter setting is compatible with ios defaults and
> looking something like:
>
> [edit interfaces ge-5/0/0]
> harry at frank# show
> mtu 1526;
> unit 0 {
> family inet {
> mtu 1500;
> address 10.1.4.1/30;
> }
> family iso;
> family mpls;
> }
>
> [edit interfaces ge-5/0/0]
> harry at frank# run show interfaces ge-5/0/0
>
> Physical interface: ge-5/0/0, Enabled, Physical link is Up
> Interface index: 173, SNMP if Index: 67
> Link-level type: Ethernet, MTU: 1526, Speed: 1000mbps,
> Loopback: Disabled, Source filtering: Disabled,
> Flow control: Enabled, Auto-negotiation: Enabled, Remote
> fault: Online
> Device flags : Present Running
> Interface flags: SNMP-Traps Internal: 0x4000
> Link flags : None
> Wavelength : 0.00 nm, Frequency: 0.00 THz
> CoS queues : 8 supported, 4 maximum usable queues
> Current address: 00:05:85:72:4d:f6, Hardware address:
> 00:05:85:72:4d:f6
> Last flapped : 2006-10-02 00:31:15 EDT (10:34:05 ago)
> Input rate : 0 bps (0 pps)
> Output rate : 0 bps (0 pps)
> Active alarms : None
> Active defects : None
>
> Logical interface ge-5/0/0.0 (Index 72) (SNMP ifIndex 90)
> Flags: SNMP-Traps Encapsulation: ENET2
> Input packets : 21
> Output packets: 26
> Protocol inet, MTU: 1500
> Flags: User-MTU
> Addresses, Flags: Is-Preferred Is-Primary
> Destination: 10.1.4.0/30, Local: 10.1.4.1, Broadcast: 10.1.4.3
> Protocol iso, MTU: 1509
> Flags: Is-Primary
> Protocol mpls, MTU: 1500
> Flags: Is-Primary
>
>
> Regards
>
>
>
>
> ________________________________
>
> From: Ihsan Junaidi Ibrahim [mailto:ihsan.junaidi at gmail.com]
> Sent: Sunday, October 01, 2006 12:12 PM
> To: Harry Reynolds
> Cc: Juniper-NSP
> Subject: Re: [j-nsp] Juniper Ethernet MTU question
>
>
> Sorry my bad. Turned out the VPN is up afterall after
> reconfiguring with default dot.1q MTU, 1518. :)
>
> Referring to your reply above, if the mpls mtu is
> always configured to be 12 bytes lesser (1488) than the
> default media mtu and the inet mtu is configured to match the
> media mtu ie: 1500, would it be correct to assume that all
> inet packets traversing through the mpls tunnel be fragmented?
>
>
> On 10/1/06, Harry Reynolds <harry at juniper.net> wrote:
>
> Can you provide a specific example of the
> error/failure you refer to
> below? I have seen an l2 circuit fail to come
> up in the control plane
> due to mtu mismatch between CEs, but never for
> mtu mismatch between
> pe/ce. These normally just results in silent
> discards; there is no
> signaling mechanism between ce/pe that would
> uncover such an mtu
> mismatch.
>
> I find our manual is quite confusing wrt to
> encap OH, so continuing to
> quote it will not do us much good. ;)
>
>
> My understanding of the system is that no
> ccc/vpn related OH is added at
> ingress by the PE. Ingress decapsulate (well,
> not so much in a L2 vpn),
> and egress encapsulates. The additional vrf
> label and mpls transport
> label, and any martini control word, are added
> at egress.
>
> On the PE router's CE facing interface encap
> setting like vlan-ccc
> simply tell the ingress circuitry what it
> should expect to pull off the
> wire, in this case a L2 frame (ccc), with a
> vlan tag (vlan).
>
>
> As further evidence, I see the same device mtu
> with just vlan-tagging,
> and with vlan-tag and vlan-ccc, which to me
> indicates there is no ccc OH
> on the CE facing interface.
>
>
> [edit interfaces fe-3/0/10]
> harry at vpn01# set vlan-tagging
>
> [edit interfaces fe-3/0/10]
> harry at vpn01 # commit
> commit complete
>
>
> [edit interfaces fe-3/0/10]
> harry at vpn01# run show interfaces fe-3/0/10
> Physical interface: fe-3/0/10, Enabled,
> Physical link is Up
> Interface index: 280, SNMP ifIndex: 103
> Link-level type: Ethernet, MTU: 1518, Speed:
> 100mbps, Loopback:
> Disabled, Source filtering: Disabled,
> Flow control: Enabled
> Device flags : Present Running
> Interface flags: SNMP-Traps Internal: 0x4000
> CoS queues : 4 supported, 4 maximum usable queues
> Current address: 00:90:69:4a:e9:84, Hardware address:
> 00:90:69:4a:e9:84
> Last flapped : 2006-09-30 03:45:10 PDT (1d
> 04:24 ago)
> Input rate : 0 bps (0 pps)
> Output rate : 0 bps (0 pps)
> Active alarms : None
> Active defects : None
> [edit interfaces fe-3/0/10]
> harry at vpn01# set encapsulation vlan-ccc
>
> [edit interfaces fe-3/0/10]
> harry at vpn01# commit
> commit complete
>
> [edit interfaces fe-3/0/10]
> harry at vpn01# run show interfaces fe-3/0/10
> Physical interface: fe-3/0/10, Enabled,
> Physical link is Up
> Interface index: 280, SNMP ifIndex: 103
> Link-level type: VLAN-CCC, MTU: 1518, Speed:
> 100mbps, Loopback:
> Disabled, Source filtering: Disabled,
> Flow control: Enabled
> Device flags : Present Running
> Interface flags: SNMP-Traps Internal: 0x4000
> CoS queues : 4 supported, 4 maximum usable queues
> Current address: 00:90:69:4a:e9:84, Hardware address:
> 00:90:69:4a:e9:84
> Last flapped : 2006-09-30 03:45:10 PDT (1d
> 04:23 ago)
> Input rate : 0 bps (0 pps)
> Output rate : 0 bps (0 pps)
> Active alarms : None
> Active defects : None
>
> Cheers
>
>
>
>
>
> ________________________________
>
> From: Ihsan Junaidi Ibrahim
> [mailto:ihsan.junaidi at gmail.com]
> Sent: Saturday, September 30, 2006 10:45 AM
> To: Harry Reynolds
> Cc: Juniper-NSP
> Subject: Re: [j-nsp] Juniper Ethernet
> MTU question
>
>
> Let's put the dfe away from being a
> core interface for a moment.
> I think I can grasp some understandings on that
> part, thanks to you :)
>
> Now back to the question no.1.
>
> My understanding is that enabling
> vlan-tagging only adds another
> 4 bytes to cater to the vlan tag. That makes
> the total mtu size 1518 and
> rightly so. From the documentation I quoted
> above, vlan-ccc (4 bytes)
> encapsulation OH is defined separately from the
> 802.1q encapsulation
> which is 18 bytes. From what I *think* I
> understand, arriving at the
> CE-facing interface, the router will append
> another 4 bytes of vlan-ccc
> encap in addition to the 1518 bytes mtu payload
> coming from the CE
> bringing it to 1522.
>
> Configuring anything less than 1522 mtu
> on the CE-facing
> interface will cause the l2vpn instance to fail
> and I couldn't ascertain
> to why. This would be the only reason why I
> otherwise would have thought
> so but am hoping someone will be able to clarify.
>
> On 10/1/06, Harry Reynolds
> <harry at juniper.net <mailto:harry at juniper.net> > wrote:
>
> If I follow, the OH added by
> the PE counts towards the
> PE-P facing (and other core) interfaces.
>
> In my diagram I tried to show that:
>
> CE facing interface with vlan
> will need 1504 + 14 or
> 1518 device mtu
>
> P facing interface will need
> that, plus (optional
> control word, vrf/mpls label, link encap).
>
> The pe does not add any
> additional vlan tags, unless you
> have vlan tagging on the p-facing interface,
> which I did not include in
> my example. I would say that the OH added by
> the PE affects the core
> facing, not CE facing side. This is why I
> started by saying the small
> jumbo supported on the DFE should be OK for a
> VRF (PE-CE) interface. As
> a core interface I think you will have issues.
>
>
>
> I hope this clarifies.
>
>
>
>
>
>
> ________________________________
>
> From: Ihsan Junaidi
> Ibrahim [mailto:
> ihsan.junaidi at gmail.com
> <mailto:ihsan.junaidi at gmail.com> ]
> Sent: Saturday,
> September 30, 2006 9:49 AM
>
> To: Harry Reynolds
> Cc: Juniper-NSP
> Subject: Re: [j-nsp]
> Juniper Ethernet MTU
> question
>
>
>
> Referring to your
> example, on the CE-facing
> interface, if i were to do a l2vpn, does the
> ccc encap OH counts against
> the interface MTU and consequently, should the
> encap overhead be counted
> against the P-facing interface? Example, the CE
> is sending 1518 bytes
> (includes vlan) and the corresponding interface
> at PE adds another 4
> bytes for vlan-ccc OH. If I follow you
> correctly that's 1522 + 8 + 14
> (ethernet core OH) = 1544 unless the 4 bytes
> ccc encap is localized to
> the CE-facing interface only and is not carried
> over to the remote
> sites.
>
>
> On 10/1/06, Harry
> Reynolds <harry at juniper.net >
> wrote:
>
> Three labels
> are typically seen in
> carrier of carrier apps. Normal
> l2/l3/vpls vpn
> is 2 labels (8 bytes),
> and as I mentioned the optional
> martini control
> word (applicable to
> l2circuits), adds another 4 bytes
> when enabled.
>
> The frame
> leaving the pe towards the p
> router will be the ethernet frame
> received from
> the CE minus preabmble/crc
> (may include a vlan tag), plus
> control word
> (optional and applicable to
> l2circuit), plus vrf label,
> plus mpls
> transport label, plus the L2
> encap needed for that link.
>
> By way of
> example consider this
> l2vpn/circuit/vpls example. AFAIK they
> all use same
> encap excepting the
> optional control world for l2 circuits:
>
> Vlan
> tagging
>
>
> cpe-------------------pe---------------p
>
> 1518
> +8 +14
>
>
vrf/mpls eth
> encap
>
> Here the PE
> must encapsulate up to 1518
> bytes of data received from CE
> (1500 + 4
> (vlan) + 14 (Eth OH). Assuming
> no control word, the pe then
> pushes two
> labels for a total payload
> size of 1526. It then encapsulates
> for link-level
> transmission to p router,
> for ethernet this adds an
> another 14
> bytes, for a grand total of
> 1540.
>
> The dfe pics
> only support a device mtu
> of 1532, so I think you have an
> issue. Loosing
> vlan tag reduces size by
> 4 bytes but the resulting 1536
> still exceeds
> dfe jumbo support.
>
> Only thing I
> can think of is to have the
> attached CEs reduce their Inet
> mtus by ~ 8
> bytes, i.e. to 1492
> including ip/tcp headers. This way the
> CEs can
> fragment as needed at ingress to
> the vpn. Not a very elegant
> solution, but
> should work.
>
> Regards
>
>
>
>
>
>
>
________________________________
>
> From:
> Ihsan Junaidi Ibrahim
> [mailto: ihsan.junaidi at gmail.com
> <mailto:ihsan.junaidi at gmail.com > ]
> Sent:
> Saturday, September 30,
> 2006 5:34 AM
> To:
> Harry Reynolds
> Cc: Juniper-NSP
>
> Subject: Re: [j-nsp] Juniper
> Ethernet MTU question
>
>
> The
> interface will be used for a
> P-facing temporarily until a
> better PIC can
> be POed. In what
> situation does an MPLS application
> warrant up to 3
> labels or an additional
> of 8 bytes over stock header of
> 4 bytes? How
> many labels does a VPLS
> transport require?
>
>
> On
> 9/29/06, Harry Reynolds <
> harry at juniper.net> wrote:
>
>
> Ah, dfe mtus. A subject
> near to my heart theses days.
>
>
> A few things.
>
>
> 1. The device mtu
> includes 14 bytes of ethernet header,
> the real mtu is
>
> 1500 bytes.
>
>
> 2. The inet mtu is
> automatically set to 1500, which
> matches real
>
ethernet mtu.
>
>
> 3. The mpls mtu is
> automatically computed to be 12 bytes
> less than real
>
> mtu to accommodate up to
> 3 labels.
>
>
> In you case it does not
> sound that inet will be
> configured. The
> vlan tag
>
> does add an extra 4
> bytes to the ethernet payload,
> resulting in need for
>
> 1504 byes of information
> and total device mtu of 1518.
> JUNOS software
>
automatically adjust the
> device mtu when you enable vlan
> tagging.
>
>
> As for support of mpls,
> I assume this is a ce facing
> interface. As such
>
packets to and from this
> interface will not be labeled.
> Labels are
>
pushed as the packet
> enters the core. In a typical l2
> vpn there would be
>
> two labels added; vrf
> and outer transport. The core
> interfaces will need
>
> an mtu of ethernet
> (1500) + vlan (4) + martini encap (0
> | 4) + labels
>
> (8) + what ever link
> encap is needed based on core
> interface type. If
>
> the core interfaces are
> also ethernet then this is
> another 14 for total
>
> of 1526-1530 (the latter
> is with optional martini
> control word). Pos
>
> only add 4 bytes for
> link encap so mtu for core would be
> at least
>
1516-1520 for pos.
>
>
> As you seem aware the
> max mtu on the dfe cards is only
> 1532. In order to
>
> get this level of jumbo
> support you need to make sure
> you have the
>
> latest fpga. See below
> for tips on that. I believe that
> you will be OK
>
> with these cards as ce
> facing with the 1518 device mtu
> that results from
>
> simply enabling vlan
> tagging, but should be safe to
> enable a larger mtu
>
> as well. Technically the
> mtu of the CE devices should be
> used to
>
dimension your ce and
> core facing interfaces, as best I
> can tell you
>
> will not need over 1518
> device mtu if ce uses default
> ethernet mtu.
>
>
> To determine if you have
> the needed rev "c" 12x dfe
> firmware, do this
>
> (note hidden commands
> should only be used with Jtac
> guidance, and I'm
>
> not in jtac):
>
>
>
Determine fpc/pic
> location. In this example its 0/2.
>
>
> Go to a shell, become
> root and vty to the fpc (0 in this
> case). Or use
>
> hidden command:
>
>
> start shell pfe network
> fpc0
>
>
> [edit interfaces
> fe-0/2/0]
>
regress at boing# run start
> shell pfe network fpc0
>
>
>
> FPC platform (PPC 603e
> processor, 32MB memory, 256KB
> flash)
>
>
FPC0(vty)#
>
>
> Now display the dfpga
> rev:
>
>
FPC0(vty)# show dfe-pic
> 2 defpga
>
>
> PIC 2 DFE information:
>
>
Dense FE PIC, 12 port(s)
> 1 GE links 1 cards
>
Periodics are enabled
>
> Normal interrupts are
> enabled, total count is 406872
>
> IFD count is 12
>
> Main Card DE FPGA:
>
Channel 0
>
Transmit Packets :
> 4397, Transmit Bytes : 0
>
Receive Packets :
> 11090, Receive Bytes : 0
>
(0x850e1080)
> DFE CPU cntl : 0x8004
>
(0x850e1082)
> DFE Version : 0x000c
> <<<<< Rev c is
>
needed to get over 1500
> mtu
>
(0x850e1086)
> DFE interrupt enable : 0x0010
>
Transmit Tagged Status:
>
>
> HTHs
>
>
>
>
>
> > -----Original
> Message-----
> > From:
> juniper-nsp-bounces at puck.nether.net
>
> > [mailto:
> juniper-nsp-bounces at puck.nether.net
> <mailto:
> juniper-nsp-bounces at puck.nether.net
> <mailto:juniper-nsp-bounces at puck.nether.net>
> <mailto:juniper-nsp-bounces at puck.nether.net> >
> ] On Behalf Of
>
> > Ihsan Junaidi Ibrahim
>
> > Sent: Friday,
> September 29, 2006 7:02 AM
>
> > To: Juniper-NSP
>
> > Subject: [j-nsp]
> Juniper Ethernet MTU question
> >
>
> > Hi all,
> >
>
> > 2 questions:
> >
>
> > 1) On a 12-port FE
> PIC, for vlan-cc encapsulation,
> should the
>
> > interface MTU be
> configured as 1518, 1522 or 1504? The
>
> > documentation below
> indicates that for vlan-ccc
>
> > encapsulation, it's a
> 4-byte field but I need to be
> certain.
> >
>
>
> http://www.junipernetworks.nl/techpubs/software/junos/junos76/
> <http://www.junipernetworks.nl/techpubs/software/junos/junos76/>
>
> <http://www.junipernetworks.nl/techpubs/software/junos/junos76/>
> >
>
>
> swconfig76-network-interfaces/html/interfaces-physical-config5
>
> .html#1085070
> >
>
> > 2) For a 12-port FE
> PIC, if I were to configure the
> maximum
>
> > physical MTU of 1532,
> will the PIC be able to
> accommodate MPLS MTU?
> >
> > --
>
> > Thank you for your
> time,
>
> > Ihsan Junaidi Ibrahim
> >
> _______________________________________________
>
> > juniper-nsp mailing
> list juniper-nsp at puck.nether.net
> <mailto:
> juniper-nsp at puck.nether.net
> <mailto:juniper-nsp at puck.nether.net > >
> >
> https://puck.nether.net/mailman/listinfo/juniper-nsp
> <
> https://puck.nether.net/mailman/listinfo/juniper-nsp
> <https://puck.nether.net/mailman/listinfo/juniper-nsp> >
> >
>
>
>
>
>
> --
> Thank
> you for your time,
> Ihsan
> Junaidi Ibrahim
>
>
>
>
>
> --
> Thank you for your time,
> Ihsan Junaidi Ibrahim
>
>
>
>
> --
> Thank you for your time,
> Ihsan Junaidi Ibrahim
>
>
>
>
>
> --
> Thank you for your time,
> Ihsan Junaidi Ibrahim
>
>
More information about the juniper-nsp
mailing list