[j-nsp] high RTT on vMX(virtio mode) interfaces

[Martin T]

On Fri, Nov 2, 2018 at 6:15 PM Martin T <m4rtntns at gmail.com> wrote:
>
> Hi!
>
> I have a simple network topology where tap interface named
> ge-0.0.1-vmx1(generated by vMX orchestration scripts) is connected to
> virbr1 bridge port 1. This ge-0.0.1-vmx1 is mapped to ge-0/0/1
> interface in Junos and has IPv4 address 10.210.0.1/24 configured.
> virbr1 has 10.210.0.2/24 configured:
>
> $ ip a sh dev virbr1
> 24: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP group default
>     link/ether fe:d6:c7:04:03:1b brd ff:ff:ff:ff:ff:ff
>     inet 10.210.0.2/24 scope global virbr1
>        valid_lft forever preferred_lft forever
>     inet6 fe80::fc06:aff:fe0e:fff1/64 scope link
>        valid_lft forever preferred_lft forever
> $
>
> When I ping the 10.210.0.2 from vMX, then the RTT is at least 20ms or
> higher. Example:
>
> root at vmx1> ping 10.210.0.2 source 10.210.0.1 count 100 rapid
> PING 10.210.0.2 (10.210.0.2): 56 data bytes
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> --- 10.210.0.2 ping statistics ---
> 100 packets transmitted, 100 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 22.045/33.365/73.901/6.391 ms
>
> root at vmx1>
>
> I don't observe such behavior when I use a physical NIC. fxp0 of vmx1
> is mapped to a physical NIC and 192.168.0.180 is a physical machine in
> my network:
>
> root at vmx1> show route 192.168.0.180 terse
>
> inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
> + = Active Route, - = Last Active, * = Both
>
> A V Destination        P Prf   Metric 1   Metric 2  Next hop        AS path
> * ? 192.168.0.0/24     D   0                       >fxp0.0
>
> root at vmx1> ping 192.168.0.180 source 192.168.0.178 count 100 rapid
> PING 192.168.0.180 (192.168.0.180): 56 data bytes
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> --- 192.168.0.180 ping statistics ---
> 100 packets transmitted, 100 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.103/0.137/0.295/0.023 ms
>
> root at vmx1>
>
> Both VCP and VFP have the lowest niceness value, load of the host
> server is low, there is no netem configuration, which might affect the
> latency, etc.
>
> Is such latency expected when using virtio mode?
>
>
> thanks,
> Martin

If it helps someone, then I do not observe the behavior described
above using vMX 18.2R1.9 on Ubuntu 16.04.5 LTS(Linux version
4.4.0-62-lowlatency) in performance mode:

root at vmx> show route 192.168.122.1 terse

inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

A V Destination        P Prf   Metric 1   Metric 2  Next hop        AS path
* ? 192.168.122.0/24   D   0                       >ge-0/0/0.0

root at vmx> ping 192.168.122.1 source 192.168.122.10 count 100 rapid
PING 192.168.122.1 (192.168.122.1): 56 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 192.168.122.1 ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.140/0.243/1.395/0.183 ms

root at vmx>

Hardware is exactly the same and I'm using virtio NICs.


Martin