<div dir="auto">The congestion might be on the reverse path. Can you get a trace route in the opposite direction?<div dir="auto"><br></div><div dir="auto">Thanks.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 4, 2022, 7:55 AM Lukas Tribus via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear list,<br>
<br>
<br>
There is major congestion for 8 - 10 hours every day on the GTT<br>
network from Frankfurt, Germany towards Milan, Italy.<br>
<br>
Before noon (localtime, that would be 1200~1300Z) things are looking<br>
fine but in the afternoon, packet-loss and latency increase steadily<br>
(peaking in the evening). We are talking packet-loss in excess of 10%<br>
and ten times the expected latency:<br>
<br>
lukas@htznr2:~$ mtr 62.10.128.1 -wrs1000 -c1000<br>
Start: 2022-02-03T21:33:28+0100<br>
<br>
HOST: htznr2 Loss% Snt Last<br>
Avg Best Wrst StDev<br>
1.|-- _gateway 0.0% 1000 3.1<br>
4.9 2.3 18.0 1.3<br>
2.|-- 10114.your-cloud.host 0.0% 1000 0.2<br>
0.3 0.2 4.0 0.2<br>
3.|-- ??? 100.0 1000 0.0<br>
0.0 0.0 0.0 0.0<br>
4.|-- <a href="http://85-10-250-153.clients.your-server.de" rel="noreferrer noreferrer" target="_blank">85-10-250-153.clients.your-server.de</a> 0.0% 1000 1.3<br>
4.8 0.8 98.6 11.8<br>
5.|-- <a href="http://spine12.cloud1.nbg1.hetzner.com" rel="noreferrer noreferrer" target="_blank">spine12.cloud1.nbg1.hetzner.com</a> 0.0% 1000 1.1<br>
4.8 0.8 66.8 11.4<br>
6.|-- <a href="http://213-239-239-121.clients.your-server.de" rel="noreferrer noreferrer" target="_blank">213-239-239-121.clients.your-server.de</a> 0.0% 1000 3.0<br>
2.3 0.5 43.4 4.1<br>
7.|-- <a href="http://core1.fra.hetzner.com" rel="noreferrer noreferrer" target="_blank">core1.fra.hetzner.com</a> 0.0% 1000 3.6<br>
5.7 3.5 65.2 5.0<br>
8.|-- <a href="http://ae18.cr6-fra2.ip4.gtt.net" rel="noreferrer noreferrer" target="_blank">ae18.cr6-fra2.ip4.gtt.net</a> 0.0% 1000 4.8<br>
5.8 3.6 116.7 6.7<br>
9.|-- <a href="http://et-5-1-0.cr0-mil2.ip4.gtt.net" rel="noreferrer noreferrer" target="_blank">et-5-1-0.cr0-mil2.ip4.gtt.net</a> 11.1% 1000 114.0<br>
112.3 63.5 172.7 13.4<br>
10.|-- <a href="http://tiscali.ip4.gtt.net" rel="noreferrer noreferrer" target="_blank">tiscali.ip4.gtt.net</a> 11.5% 1000 111.5<br>
113.4 64.3 148.8 13.2<br>
11.|-- <a href="http://static-94-32-126-81.clienti.tiscali.it" rel="noreferrer noreferrer" target="_blank">static-94-32-126-81.clienti.tiscali.it</a> 10.6% 1000 121.7<br>
126.4 79.8 162.9 13.7<br>
12.|-- <a href="http://static-94-32-135-162.clienti.tiscali.it" rel="noreferrer noreferrer" target="_blank">static-94-32-135-162.clienti.tiscali.it</a> 11.2% 1000 119.1<br>
126.8 80.9 176.8 13.9<br>
13.|-- <a href="http://static-94-32-135-182.clienti.tiscali.it" rel="noreferrer noreferrer" target="_blank">static-94-32-135-182.clienti.tiscali.it</a> 11.3% 1000 119.0<br>
127.6 79.5 171.2 14.2<br>
14.|-- <a href="http://static-213-205-54-210.clienti.tiscali.it" rel="noreferrer noreferrer" target="_blank">static-213-205-54-210.clienti.tiscali.it</a> 78.7% 1000 130.1<br>
125.3 73.8 228.1 16.2<br>
15.|-- ??? 100.0 1000 0.0<br>
0.0 0.0 0.0 0.0<br>
lukas@htznr2:~$<br>
<br>
<br>
<br>
2022-02-01T18:10:07Z<br>
<a href="https://pastebin.com/raw/MSvuSuR0" rel="noreferrer noreferrer" target="_blank">https://pastebin.com/raw/MSvuSuR0</a><br>
<br>
2022-02-02T17:39:44Z<br>
<a href="https://pastebin.com/raw/zWD8GHGa" rel="noreferrer noreferrer" target="_blank">https://pastebin.com/raw/zWD8GHGa</a><br>
<br>
2022-02-03T20:33:28Z (as above)<br>
<a href="https://pastebin.com/raw/DHWDyi1g" rel="noreferrer noreferrer" target="_blank">https://pastebin.com/raw/DHWDyi1g</a><br>
<br>
2022-02-04T13:11:51Z<br>
<a href="https://pastebin.com/raw/qhrWZBQX" rel="noreferrer noreferrer" target="_blank">https://pastebin.com/raw/qhrWZBQX</a><br>
<br>
<br>
GTT NOC has been advised Feb 1st, 18:000Z, but apparently they always<br>
verify network issues in the morning, which is also when those links<br>
are uncongested, so at this point there is no confirmation from them.<br>
I'm assuming the network teams are aware of it anyway, but we don't<br>
know for sure. If you are using GTT for transit in Europe and traffic<br>
to Italy matters to your customers, you probably want to reroute (for<br>
example: reducing loc-pref based on GTT's BGP community 3257:53900 for<br>
prefixes learned in Italy).<br>
<br>
<br>
cheers,<br>
lukas<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank" rel="noreferrer">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>