[j-nsp] What exactly causes inconsistent RTT seen using ping utility in Junos?

[Martin T]

Hi,

ping utility(or "icmp-ping" RPM probe without hardware-timestamp) in
Junos shows occasional high RTT even in case of pinging for example
directly connected devices in the LAN. Example:

64 bytes from 10.55.55.1: icmp_seq=40 ttl=64 time=0.441 ms
64 bytes from 10.55.55.1: icmp_seq=41 ttl=64 time=0.399 ms
64 bytes from 10.55.55.1: icmp_seq=42 ttl=64 time=67.192 ms
64 bytes from 10.55.55.1: icmp_seq=43 ttl=64 time=0.617 ms

This is well know behavior and documented in several KB articles.
However, what exactly causes this?

KB11792 says that it is caused by an internal scheduler in router
which gives lowest priority to ICMP ping packets. KB28157 gives an
impression that it has something to do with default CoS configuration.
When I measure the deltas between ICMP "echo request" messages seen by
the server directly connected to the router, then there is always a
longer delay for the delayed ICMP "echo request" message. For example,
here is the packet capture with delays between ICMP "echo request"
messages for those four ping results shows above:

 00:00:01.000395 IP 10.55.55.2 > 10.55.55.1: ICMP echo request, id
59201, seq 40, length 64
 00:00:01.000218 IP 10.55.55.2 > 10.55.55.1: ICMP echo request, id
59201, seq 41, length 64
 00:00:01.024137 IP 10.55.55.2 > 10.55.55.1: ICMP echo request, id
59201, seq 42, length 64
 00:00:00.983721 IP 10.55.55.2 > 10.55.55.1: ICMP echo request, id
59201, seq 43, length 64

Packet 42 arrives later than normal(normal is ~1000ms) and thus next
packet arrives earlier than normal. This means that (at least some of)
this occasional abnormal delay should be introduced when ICMP "echo
request" is traveling out of the box.

Based on this information I changed the CoS configuration in two MX
series routers. First I tried with vMX(runs in performance mode) and
configured RE-sent ICMP "echo request" messages into forwarding-class
named "ef" and associated this class with low-latency queue:

root at vmx1> show configuration class-of-service
forwarding-classes {
    class ef queue-num 1 priority high policing-priority premium;
    class be queue-num 0 priority low policing-priority normal;
}
interfaces {
    ge-0/0/9 {
        scheduler-map test_schedmap;
    }
}
scheduler-maps {
    test_schedmap {
        forwarding-class be scheduler be_sched;
        forwarding-class ef scheduler ef_sched;
    }
}
schedulers {
    be_sched {
        priority low;
    }
    ef_sched {
        buffer-size percent 0;
        priority strict-high;
    }
}

root at vmx1>

I can confirm that ICMP "echo request" messages are put into queue
number 1(associated with "ef" forwarding-class) with "show interfaces
queue ge-0/0/9 forwarding-class ef" command. However, it made no
difference compared to default CoS configuration where ICMP "echo
request" messages sent by RE are in the best-effort class. When I try
the same configuration on MX960, then again, there is no difference
compared to default CoS configuration.

I also noticed that when I load the RE CPU, then this does not have
affect on ping RTT. However, for example, when I pin the vPFE vCPU's
to specific CPU core and then additionally do something CPU-heavy on
that specific core, then it has noticeable affect on RTT of RE-sent
ICMP "echo request" messages.


Maybe someone (from Juniper) can shed some light on it. Please let me
know if anything was unclear.


thanks,
Martin