[outages] AS3257 GTT latency/packet loss issues between Stockholm and Frankfurt

[Tore Anderson]

I first reported this to inoc at gt-t.net, but they refused to look into
it, as I'm not a customer of theirs. (An attitude which makes me very
happy I'm not, incidentally.). So I'll dump the info here in case
someone a more senior techie in GTT is reading this list and would like
to take a closer look.

Traffic from GTT's Stockholm PoP (in my case delivered to them there by
AS3549 to 77.67.82.145 a.k.a xe-5-2-0.stk30.ip4.tinet.net) destined
to/past their Frankfurt PoP gets either routed to 141.136.107.218
(ae5.fra61.ip4.tinet.net) or 89.149.185.105
(xe-5-0-1.fra61.ip4.tinet.net). It appears they're using ECMP between
these two links, as I can "choose" by slightly modifying the source
and/or destination address when I'm tracing (even though the addresses
remain within the same DFZ prefix).

When the path to 141.136.107.218 is chosen, the RTT increases by ~100ms
and there is significant packet loss. See below traceroutes, which are
done at the same time from the same source.

The return path is known to be good. If I change my outbound path to
the destinations I'm tracing to below in AS12741 to go via AS3356->AS174,
the problem immediately disappears. So for now I'm strong depreffing all
as-paths matching .*3257.* and I'm happy, but I'm thinking someone else
might be interested in looking into this...

Good path (xe-5-2-0.stk30. -> xe-5-0-1.fra61.ip4.tinet.net):
------------------------------------------------------------
$ sudo mtr -nrc 100 5.226.119.193
pompel.hmg9.vgnett.no             Snt: 100   Loss%  Last   Avg  Best  Wrst StDev
87.238.51.121                                 0.0%   0.9   1.2   0.8   4.1   0.5
87.238.62.80                                  0.0%   1.3   1.4   0.8   5.4   0.8
87.238.62.84                                  0.0%   0.3   1.3   0.3  50.2   5.4
64.210.69.85                                  0.0%   0.5   6.4   0.5 200.8  26.5
67.16.136.6                                   0.0%  14.1  13.8   7.3  43.8   3.4
67.17.74.46                                   0.0%   7.6   7.6   7.5  12.9   0.6
77.67.82.145                                  0.0%   7.5  11.9   7.5  70.0  12.1
89.149.185.105                                0.0%  30.0  30.8  30.0  71.8   4.8
77.67.73.78                                   0.0%  47.8  48.7  46.7  50.7   1.2
83.238.250.44                                 0.0%  54.8  55.6  53.9  57.2   1.0
83.238.251.217                                0.0%  55.2  56.0  54.6  60.3   1.1
81.210.5.70                                   0.0%  54.1  54.3  53.9  59.4   0.6
5.226.119.193                                 0.0%  59.0  58.4  57.0  67.9   1.7

Bad path (xe-5-2-0.stk30. -> ae5.fra61.ip4.tinet.net):
------------------------------------------------------
$ sudo mtr -nrc 100 5.226.119.170
pompel.hmg9.vgnett.no             Snt: 100   Loss%  Last   Avg  Best  Wrst StDev
87.238.51.121                                 0.0%   1.0   1.0   0.8   4.0   0.3
87.238.62.80                                  0.0%   1.7   1.5   0.8   7.5   1.0
87.238.62.84                                  0.0%   0.3   2.2   0.3  66.7   8.4
64.210.69.85                                  0.0%   0.5   4.6   0.5 153.4  17.9
67.16.146.93                                  0.0%   7.3   7.4   7.3  21.5   1.4
67.17.74.42                                   0.0%  16.9  15.9   7.5  17.0   1.1
77.67.82.145                                  0.0%   7.5   9.8   7.5  66.9   9.6
141.136.107.218                               7.0% 153.6 152.7 128.2 182.4   5.3
77.67.75.38                                  12.0% 156.6 154.9 140.2 163.4   4.0
83.238.251.98                                 8.0% 157.7 157.1 143.6 176.5   5.8
83.238.250.5                                  6.0% 163.4 162.5 150.0 175.3   4.9
77.252.5.118                                  3.0% 159.0 162.2 150.6 226.0  14.5
5.226.126.223                                 7.0% 157.0 161.0 149.6 223.8  12.6
5.226.119.170                                 5.0% 153.5 161.2 147.7 226.5  11.4

Tore