[outages] GTT daily congestion in Europe

[Lukas Tribus]

Dear list,


There is major congestion for 8 - 10 hours every day on the GTT
network from Frankfurt, Germany towards Milan, Italy.

Before noon (localtime, that would be 1200~1300Z) things are looking
fine but in the afternoon, packet-loss and latency increase steadily
(peaking in the evening). We are talking packet-loss in excess of 10%
and ten times the expected latency:

lukas at htznr2:~$ mtr 62.10.128.1 -wrs1000 -c1000
Start: 2022-02-03T21:33:28+0100

HOST: htznr2                                   Loss%   Snt   Last
Avg  Best  Wrst StDev
  1.|-- _gateway                                  0.0%  1000    3.1
4.9   2.3  18.0   1.3
  2.|-- 10114.your-cloud.host                     0.0%  1000    0.2
0.3   0.2   4.0   0.2
  3.|-- ???                                      100.0  1000    0.0
0.0   0.0   0.0   0.0
  4.|-- 85-10-250-153.clients.your-server.de      0.0%  1000    1.3
4.8   0.8  98.6  11.8
  5.|-- spine12.cloud1.nbg1.hetzner.com           0.0%  1000    1.1
4.8   0.8  66.8  11.4
  6.|-- 213-239-239-121.clients.your-server.de    0.0%  1000    3.0
2.3   0.5  43.4   4.1
  7.|-- core1.fra.hetzner.com                     0.0%  1000    3.6
5.7   3.5  65.2   5.0
  8.|-- ae18.cr6-fra2.ip4.gtt.net                 0.0%  1000    4.8
5.8   3.6 116.7   6.7
  9.|-- et-5-1-0.cr0-mil2.ip4.gtt.net            11.1%  1000  114.0
112.3  63.5 172.7  13.4
 10.|-- tiscali.ip4.gtt.net                      11.5%  1000  111.5
113.4  64.3 148.8  13.2
 11.|-- static-94-32-126-81.clienti.tiscali.it   10.6%  1000  121.7
126.4  79.8 162.9  13.7
 12.|-- static-94-32-135-162.clienti.tiscali.it  11.2%  1000  119.1
126.8  80.9 176.8  13.9
 13.|-- static-94-32-135-182.clienti.tiscali.it  11.3%  1000  119.0
127.6  79.5 171.2  14.2
 14.|-- static-213-205-54-210.clienti.tiscali.it 78.7%  1000  130.1
125.3  73.8 228.1  16.2
 15.|-- ???                                      100.0  1000    0.0
0.0   0.0   0.0   0.0
lukas at htznr2:~$



2022-02-01T18:10:07Z
https://pastebin.com/raw/MSvuSuR0

2022-02-02T17:39:44Z
https://pastebin.com/raw/zWD8GHGa

2022-02-03T20:33:28Z (as above)
https://pastebin.com/raw/DHWDyi1g

2022-02-04T13:11:51Z
https://pastebin.com/raw/qhrWZBQX


GTT NOC has been advised Feb 1st, 18:000Z, but apparently they always
verify network issues in the morning, which is also when those links
are uncongested, so at this point there is no confirmation from them.
I'm assuming the network teams are aware of it anyway, but we don't
know for sure. If you are using GTT for transit in Europe and traffic
to Italy matters to your customers, you probably want to reroute (for
example: reducing loc-pref based on GTT's BGP community 3257:53900 for
prefixes learned in Italy).


cheers,
lukas