[j-nsp] Juniper Ethernet MTU question
Harry Reynolds
harry at juniper.net
Mon Oct 2 11:08:48 EDT 2006
hehe, Now you go and open a can of worms. ;)
Transit LSRs are incapable of performing fragmentation; therefore there
is potential for resulting discard anytime mtus are mismatched.
Our default ethernet mtu settings can lead to fragmentation, as you
infer, when we are acting as an ingress node for transit or locally
generated traffic. Here is proof of this behavior:
[edit protocols mpls]
harry at frank# run ping 10.3.4.1 size 1460 count 1
PING 10.3.4.1 (10.3.4.1): 1460 data bytes
1468 bytes from 10.3.4.1: icmp_seq=0 ttl=64 time=1.932 ms
--- 10.3.4.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.932/1.932/1.932/0.000 ms
11:02:05.828893 Out MPLS (label 100000, exp 0, [S], ttl 64), IP, length:
1492
[edit protocols mpls]
harry at frank# run ping 10.3.4.1 size 1461 count 1
PING 10.3.4.1 (10.3.4.1): 1461 data bytes
1469 bytes from 10.3.4.1: icmp_seq=0 ttl=64 time=20.195 ms
--- 10.3.4.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 20.195/20.195/20.195/0.000 ms
11:02:10.323250 Out MPLS (label 100000, exp 0, [S], ttl 63), IP, length:
1488
11:02:10.323260 Out MPLS (label 100000, exp 0, [S], ttl 63), IP, length:
29
As I understand this behavior differs from cisco defaults, which IIRC
would have generated a jumbo in this case (ios adds up to 3 labels on
top of device mtu, rather than reducing payload size to accommodate
labels).
Two options present themselves. If you do not want jumbos (or
fragmentation) set inet mtu to equal mpls mtu (network wide) and take
efficiency hit associated with smaller packets. Or if you can handle
jumbos raise media mtu by 12 while forcing inet to stay at 1500 (it will
automatically adjust when device mtu is changed and interface is
bounced). Either way you end up with an inet *PDU* (which includes IP
and TCP headers), that matches the underlying mpls *MTU* (which *does
not* include the mpls headers), and there is no need for ingress
fragmentation or potential for mtu related discards at transit nodes.
The latter setting is compatible with ios defaults and looking something
like:
[edit interfaces ge-5/0/0]
harry at frank# show
mtu 1526;
unit 0 {
family inet {
mtu 1500;
address 10.1.4.1/30;
}
family iso;
family mpls;
}
[edit interfaces ge-5/0/0]
harry at frank# run show interfaces ge-5/0/0
Physical interface: ge-5/0/0, Enabled, Physical link is Up
Interface index: 173, SNMP if Index: 67
Link-level type: Ethernet, MTU: 1526, Speed: 1000mbps, Loopback:
Disabled, Source filtering: Disabled,
Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
Wavelength : 0.00 nm, Frequency: 0.00 THz
CoS queues : 8 supported, 4 maximum usable queues
Current address: 00:05:85:72:4d:f6, Hardware address:
00:05:85:72:4d:f6
Last flapped : 2006-10-02 00:31:15 EDT (10:34:05 ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
Logical interface ge-5/0/0.0 (Index 72) (SNMP ifIndex 90)
Flags: SNMP-Traps Encapsulation: ENET2
Input packets : 21
Output packets: 26
Protocol inet, MTU: 1500
Flags: User-MTU
Addresses, Flags: Is-Preferred Is-Primary
Destination: 10.1.4.0/30, Local: 10.1.4.1, Broadcast: 10.1.4.3
Protocol iso, MTU: 1509
Flags: Is-Primary
Protocol mpls, MTU: 1500
Flags: Is-Primary
Regards
________________________________
From: Ihsan Junaidi Ibrahim [mailto:ihsan.junaidi at gmail.com]
Sent: Sunday, October 01, 2006 12:12 PM
To: Harry Reynolds
Cc: Juniper-NSP
Subject: Re: [j-nsp] Juniper Ethernet MTU question
Sorry my bad. Turned out the VPN is up afterall after
reconfiguring with default dot.1q MTU, 1518. :)
Referring to your reply above, if the mpls mtu is always
configured to be 12 bytes lesser (1488) than the default media mtu and
the inet mtu is configured to match the media mtu ie: 1500, would it be
correct to assume that all inet packets traversing through the mpls
tunnel be fragmented?
On 10/1/06, Harry Reynolds <harry at juniper.net> wrote:
Can you provide a specific example of the error/failure
you refer to
below? I have seen an l2 circuit fail to come up in the
control plane
due to mtu mismatch between CEs, but never for mtu
mismatch between
pe/ce. These normally just results in silent discards;
there is no
signaling mechanism between ce/pe that would uncover
such an mtu
mismatch.
I find our manual is quite confusing wrt to encap OH, so
continuing to
quote it will not do us much good. ;)
My understanding of the system is that no ccc/vpn
related OH is added at
ingress by the PE. Ingress decapsulate (well, not so
much in a L2 vpn),
and egress encapsulates. The additional vrf label and
mpls transport
label, and any martini control word, are added at
egress.
On the PE router's CE facing interface encap setting
like vlan-ccc
simply tell the ingress circuitry what it should expect
to pull off the
wire, in this case a L2 frame (ccc), with a vlan tag
(vlan).
As further evidence, I see the same device mtu with just
vlan-tagging,
and with vlan-tag and vlan-ccc, which to me indicates
there is no ccc OH
on the CE facing interface.
[edit interfaces fe-3/0/10]
harry at vpn01# set vlan-tagging
[edit interfaces fe-3/0/10]
harry at vpn01 # commit
commit complete
[edit interfaces fe-3/0/10]
harry at vpn01# run show interfaces fe-3/0/10
Physical interface: fe-3/0/10, Enabled, Physical link is
Up
Interface index: 280, SNMP ifIndex: 103
Link-level type: Ethernet, MTU: 1518, Speed: 100mbps,
Loopback:
Disabled, Source filtering: Disabled,
Flow control: Enabled
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
CoS queues : 4 supported, 4 maximum usable queues
Current address: 00:90:69:4a:e9:84, Hardware address:
00:90:69:4a:e9:84
Last flapped : 2006-09-30 03:45:10 PDT (1d 04:24
ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
[edit interfaces fe-3/0/10]
harry at vpn01# set encapsulation vlan-ccc
[edit interfaces fe-3/0/10]
harry at vpn01# commit
commit complete
[edit interfaces fe-3/0/10]
harry at vpn01# run show interfaces fe-3/0/10
Physical interface: fe-3/0/10, Enabled, Physical link is
Up
Interface index: 280, SNMP ifIndex: 103
Link-level type: VLAN-CCC, MTU: 1518, Speed: 100mbps,
Loopback:
Disabled, Source filtering: Disabled,
Flow control: Enabled
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
CoS queues : 4 supported, 4 maximum usable queues
Current address: 00:90:69:4a:e9:84, Hardware address:
00:90:69:4a:e9:84
Last flapped : 2006-09-30 03:45:10 PDT (1d 04:23
ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
Cheers
________________________________
From: Ihsan Junaidi Ibrahim
[mailto:ihsan.junaidi at gmail.com]
Sent: Saturday, September 30, 2006 10:45 AM
To: Harry Reynolds
Cc: Juniper-NSP
Subject: Re: [j-nsp] Juniper Ethernet MTU
question
Let's put the dfe away from being a core
interface for a moment.
I think I can grasp some understandings on that part,
thanks to you :)
Now back to the question no.1.
My understanding is that enabling vlan-tagging
only adds another
4 bytes to cater to the vlan tag. That makes the total
mtu size 1518 and
rightly so. From the documentation I quoted above,
vlan-ccc (4 bytes)
encapsulation OH is defined separately from the 802.1q
encapsulation
which is 18 bytes. From what I *think* I understand,
arriving at the
CE-facing interface, the router will append another 4
bytes of vlan-ccc
encap in addition to the 1518 bytes mtu payload coming
from the CE
bringing it to 1522.
Configuring anything less than 1522 mtu on the
CE-facing
interface will cause the l2vpn instance to fail and I
couldn't ascertain
to why. This would be the only reason why I otherwise
would have thought
so but am hoping someone will be able to clarify.
On 10/1/06, Harry Reynolds <harry at juniper.net
<mailto:harry at juniper.net> > wrote:
If I follow, the OH added by the PE
counts towards the
PE-P facing (and other core) interfaces.
In my diagram I tried to show that:
CE facing interface with vlan will need
1504 + 14 or
1518 device mtu
P facing interface will need that, plus
(optional
control word, vrf/mpls label, link encap).
The pe does not add any additional vlan
tags, unless you
have vlan tagging on the p-facing interface, which I did
not include in
my example. I would say that the OH added by the PE
affects the core
facing, not CE facing side. This is why I started by
saying the small
jumbo supported on the DFE should be OK for a VRF
(PE-CE) interface. As
a core interface I think you will have issues.
I hope this clarifies.
________________________________
From: Ihsan Junaidi Ibrahim
[mailto:
ihsan.junaidi at gmail.com <mailto:ihsan.junaidi at gmail.com>
]
Sent: Saturday, September 30,
2006 9:49 AM
To: Harry Reynolds
Cc: Juniper-NSP
Subject: Re: [j-nsp] Juniper
Ethernet MTU
question
Referring to your example, on
the CE-facing
interface, if i were to do a l2vpn, does the ccc encap
OH counts against
the interface MTU and consequently, should the encap
overhead be counted
against the P-facing interface? Example, the CE is
sending 1518 bytes
(includes vlan) and the corresponding interface at PE
adds another 4
bytes for vlan-ccc OH. If I follow you correctly that's
1522 + 8 + 14
(ethernet core OH) = 1544 unless the 4 bytes ccc encap
is localized to
the CE-facing interface only and is not carried over to
the remote
sites.
On 10/1/06, Harry Reynolds
<harry at juniper.net >
wrote:
Three labels are
typically seen in
carrier of carrier apps. Normal
l2/l3/vpls vpn is 2
labels (8 bytes),
and as I mentioned the optional
martini control word
(applicable to
l2circuits), adds another 4 bytes
when enabled.
The frame leaving the pe
towards the p
router will be the ethernet frame
received from the CE
minus preabmble/crc
(may include a vlan tag), plus
control word (optional
and applicable to
l2circuit), plus vrf label,
plus mpls transport
label, plus the L2
encap needed for that link.
By way of example
consider this
l2vpn/circuit/vpls example. AFAIK they
all use same encap
excepting the
optional control world for l2 circuits:
Vlan
tagging
cpe-------------------pe---------------p
1518 +8
+14
vrf/mpls eth
encap
Here the PE must
encapsulate up to 1518
bytes of data received from CE
(1500 + 4 (vlan) + 14
(Eth OH). Assuming
no control word, the pe then
pushes two labels for a
total payload
size of 1526. It then encapsulates
for link-level
transmission to p router,
for ethernet this adds an
another 14 bytes, for a
grand total of
1540.
The dfe pics only
support a device mtu
of 1532, so I think you have an
issue. Loosing vlan tag
reduces size by
4 bytes but the resulting 1536
still exceeds dfe jumbo
support.
Only thing I can think
of is to have the
attached CEs reduce their Inet
mtus by ~ 8 bytes, i.e.
to 1492
including ip/tcp headers. This way the
CEs can fragment as
needed at ingress to
the vpn. Not a very elegant
solution, but should
work.
Regards
________________________________
From: Ihsan
Junaidi Ibrahim
[mailto: ihsan.junaidi at gmail.com
<mailto:ihsan.junaidi at gmail.com > ]
Sent: Saturday,
September 30,
2006 5:34 AM
To: Harry
Reynolds
Cc: Juniper-NSP
Subject: Re:
[j-nsp] Juniper
Ethernet MTU question
The interface
will be used for a
P-facing temporarily until a
better PIC can be POed.
In what
situation does an MPLS application
warrant up to 3 labels
or an additional
of 8 bytes over stock header of
4 bytes? How many labels
does a VPLS
transport require?
On 9/29/06,
Harry Reynolds <
harry at juniper.net> wrote:
Ah, dfe
mtus. A subject
near to my heart theses days.
A few
things.
1. The
device mtu
includes 14 bytes of ethernet header,
the real mtu is
1500
bytes.
2. The
inet mtu is
automatically set to 1500, which
matches real
ethernet
mtu.
3. The
mpls mtu is
automatically computed to be 12 bytes
less than real
mtu to
accommodate up to
3 labels.
In you
case it does not
sound that inet will be
configured. The vlan tag
does add
an extra 4
bytes to the ethernet payload,
resulting in need for
1504
byes of information
and total device mtu of 1518.
JUNOS software
automatically adjust the
device mtu when you enable vlan
tagging.
As for
support of mpls,
I assume this is a ce facing
interface. As such
packets
to and from this
interface will not be labeled.
Labels are
pushed
as the packet
enters the core. In a typical l2
vpn there would be
two
labels added; vrf
and outer transport. The core
interfaces will need
an mtu
of ethernet
(1500) + vlan (4) + martini encap (0
| 4) + labels
(8) +
what ever link
encap is needed based on core
interface type. If
the core
interfaces are
also ethernet then this is
another 14 for total
of
1526-1530 (the latter
is with optional martini
control word). Pos
only add
4 bytes for
link encap so mtu for core would be
at least
1516-1520 for pos.
As you
seem aware the
max mtu on the dfe cards is only
1532. In order to
get this
level of jumbo
support you need to make sure
you have the
latest
fpga. See below
for tips on that. I believe that
you will be OK
with
these cards as ce
facing with the 1518 device mtu
that results from
simply
enabling vlan
tagging, but should be safe to
enable a larger mtu
as well.
Technically the
mtu of the CE devices should be
used to
dimension your ce and
core facing interfaces, as best I
can tell you
will not
need over 1518
device mtu if ce uses default
ethernet mtu.
To
determine if you have
the needed rev "c" 12x dfe
firmware, do this
(note
hidden commands
should only be used with Jtac
guidance, and I'm
not in
jtac):
Determine fpc/pic
location. In this example its 0/2.
Go to a
shell, become
root and vty to the fpc (0 in this
case). Or use
hidden
command:
start
shell pfe network
fpc0
[edit
interfaces
fe-0/2/0]
regress at boing# run start
shell pfe network fpc0
FPC
platform (PPC 603e
processor, 32MB memory, 256KB
flash)
FPC0(vty)#
Now
display the dfpga
rev:
FPC0(vty)# show dfe-pic
2 defpga
PIC 2
DFE information:
Dense FE
PIC, 12 port(s)
1 GE links 1 cards
Periodics are enabled
Normal
interrupts are
enabled, total count is 406872
IFD
count is 12
Main
Card DE FPGA:
Channel
0
Transmit Packets :
4397, Transmit Bytes : 0
Receive Packets :
11090, Receive Bytes : 0
(0x850e1080)
DFE CPU cntl : 0x8004
(0x850e1082)
DFE Version : 0x000c
<<<<< Rev c is
needed
to get over 1500
mtu
(0x850e1086)
DFE interrupt enable : 0x0010
Transmit
Tagged Status:
HTHs
>
-----Original
Message-----
> From:
juniper-nsp-bounces at puck.nether.net
>
[mailto:
juniper-nsp-bounces at puck.nether.net
<mailto:
juniper-nsp-bounces at puck.nether.net
<mailto:juniper-nsp-bounces at puck.nether.net>
<mailto:juniper-nsp-bounces at puck.nether.net> > ] On
Behalf Of
> Ihsan
Junaidi Ibrahim
> Sent:
Friday,
September 29, 2006 7:02 AM
> To:
Juniper-NSP
>
Subject: [j-nsp]
Juniper Ethernet MTU question
>
> Hi
all,
>
> 2
questions:
>
> 1) On
a 12-port FE
PIC, for vlan-cc encapsulation,
should the
>
interface MTU be
configured as 1518, 1522 or 1504? The
>
documentation below
indicates that for vlan-ccc
>
encapsulation, it's a
4-byte field but I need to be
certain.
>
http://www.junipernetworks.nl/techpubs/software/junos/junos76/
<http://www.junipernetworks.nl/techpubs/software/junos/junos76/>
<http://www.junipernetworks.nl/techpubs/software/junos/junos76/>
>
swconfig76-network-interfaces/html/interfaces-physical-config5
.html#1085070
>
> 2) For
a 12-port FE
PIC, if I were to configure the
maximum
>
physical MTU of 1532,
will the PIC be able to
accommodate MPLS MTU?
>
> --
> Thank
you for your
time,
> Ihsan
Junaidi Ibrahim
>
_______________________________________________
>
juniper-nsp mailing
list juniper-nsp at puck.nether.net
<mailto:
juniper-nsp at puck.nether.net
<mailto:juniper-nsp at puck.nether.net > >
>
https://puck.nether.net/mailman/listinfo/juniper-nsp
< https://puck.nether.net/mailman/listinfo/juniper-nsp
<https://puck.nether.net/mailman/listinfo/juniper-nsp> >
>
--
Thank you for
your time,
Ihsan Junaidi
Ibrahim
--
Thank you for your time,
Ihsan Junaidi Ibrahim
--
Thank you for your time,
Ihsan Junaidi Ibrahim
--
Thank you for your time,
Ihsan Junaidi Ibrahim
More information about the juniper-nsp
mailing list