[Outages-discussion] [outages] AT&T possible routing loop
Katie Holly
40xwol3g at meo.ws
Sat Aug 19 18:18:20 EDT 2017
We're seeing the same behavior from our looking glass servers in the us-east-1, us-east-2 and ca-central-1 AWS regions:
https://mtr.sh/#trace/4Smfr,cdl4e,ZRp_R/107.152.25.197
Traceroute from AWS AS14618 in United States, Columbus (us-east-1)
traceroute to 107.152.25.197 (107.152.25.197), 50 hops max, 60 byte packets
1 216.182.226.8 (216.182.226.8) [AS14618] 13.353 ms 216.182.226.10 (216.182.226.10) [AS14618] 53.712 ms 216.182.224.100 (216.182.224.100) [AS14618] 8.727 ms 216.182.224.34 (216.182.224.34) [AS14618] 14.511 ms 216.182.224.38 (216.182.224.38) [AS14618] 18.520 ms
2 100.66.12.194 (100.66.12.194) [*] 13.873 ms 100.66.13.152 (100.66.13.152) [*] 21.875 ms 100.66.8.152 (100.66.8.152) [*] 15.711 ms 100.66.8.148 (100.66.8.148) [*] 21.858 ms 100.66.12.106 (100.66.12.106) [*] 18.080 ms
3 100.66.15.140 (100.66.15.140) [*] 12.015 ms 100.66.15.232 (100.66.15.232) [*] 17.456 ms 100.66.10.64 (100.66.10.64) [*] 17.055 ms 100.66.11.142 (100.66.11.142) [*] 30.363 ms 100.66.11.38 (100.66.11.38) [*] 21.827 ms
4 100.66.7.133 (100.66.7.133) [*] 14.418 ms 100.66.6.247 (100.66.6.247) [*] 14.528 ms 100.66.6.157 (100.66.6.157) [*] 21.156 ms 100.66.6.129 (100.66.6.129) [*] 11.556 ms 100.66.6.135 (100.66.6.135) [*] 9.845 ms
5 100.66.4.253 (100.66.4.253) [*] 21.158 ms 100.66.4.71 (100.66.4.71) [*] 21.566 ms 100.66.4.85 (100.66.4.85) [*] 15.166 ms 100.66.4.213 (100.66.4.213) [*] 19.059 ms 100.66.4.71 (100.66.4.71) [*] 21.547 ms
6 100.65.10.129 (100.65.10.129) [*] 0.418 ms 100.65.8.129 (100.65.8.129) [*] 0.411 ms 100.65.11.129 (100.65.11.129) [*] 0.409 ms 100.65.10.1 (100.65.10.1) [*] 0.354 ms 100.65.11.65 (100.65.11.65) [*] 0.396 ms
7 205.251.244.64 (205.251.244.64) [AS16509] 2.609 ms 205.251.245.186 (205.251.245.186) [AS16509] 1.864 ms 205.251.245.225 (205.251.245.225) [AS16509] 0.797 ms 205.251.244.68 (205.251.244.68) [AS16509] 2.615 ms 52.93.24.108 (52.93.24.108) [AS16509] 2.649 ms
8 52.93.24.143 (52.93.24.143) [AS16509] 0.742 ms 52.93.24.141 (52.93.24.141) [AS16509] 0.726 ms 54.239.108.94 (54.239.108.94) [AS16509] 1.355 ms 54.239.111.4 (54.239.111.4) [AS16509] 1.844 ms 52.93.24.117 (52.93.24.117) [AS16509] 0.696 ms
9 54.239.108.94 (54.239.108.94) [AS16509] 1.342 ms 54.239.111.14 (54.239.111.14) [AS16509] 27.541 ms 54.239.108.196 (54.239.108.196) [AS16509] 19.844 ms 54.239.108.128 (54.239.108.128) [AS16509] 8.538 ms 54.239.111.8 (54.239.111.8) [AS16509] 19.837 ms
10 54.239.108.209 (54.239.108.209) [AS16509] 1.420 ms 206.121.24.33 (206.121.24.33) [*] 2.119 ms 54.239.111.235 (54.239.111.235) [AS16509] 1.150 ms 12.247.95.9 (12.247.95.9) [AS7018] 1.750 ms 54.239.111.255 (54.239.111.255) [AS16509] 0.845 ms
11 206.121.24.33 (206.121.24.33) [*] 2.070 ms 206.121.24.41 (206.121.24.41) [*] 2.277 ms 12.247.95.5 (12.247.95.5) [AS7018] 1.765 ms 206.121.24.33 (206.121.24.33) [*] 2.100 ms 12.247.95.5 (12.247.95.5) [AS7018] 1.841 ms
12 12.122.135.214 (12.122.135.214) [AS7018] 71.020 ms 12.122.135.226 (12.122.135.226) [AS7018] 66.895 ms cr2.wswdc.ip.att.net (12.122.134.94) [AS7018] 68.964 ms 12.122.135.214 (12.122.135.214) [AS7018] 74.687 ms cr2.wswdc.ip.att.net (12.122.134.94) [AS7018] 68.943 ms
13 cr2.wswdc.ip.att.net (12.122.134.94) [AS7018] 68.948 ms cr1.cgcil.ip.att.net (12.122.18.21) [AS7018] 70.113 ms cr2.wswdc.ip.att.net (12.122.134.94) [AS7018] 68.933 ms 68.926 ms 68.924 ms
14 cr1.cgcil.ip.att.net (12.122.18.21) [AS7018] 70.083 ms 69.877 ms 69.858 ms 69.843 ms 69.845 ms
15 cr2.dvmco.ip.att.net (12.122.28.77) [AS7018] 68.379 ms 68.819 ms 68.819 ms 68.805 ms 68.821 ms
16 cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 64.733 ms cr1.slkut.ip.att.net (12.122.30.25) [AS7018] 68.766 ms cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 64.667 ms cr1.slkut.ip.att.net (12.122.30.25) [AS7018] 68.760 ms cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 64.667 ms
17 cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 64.653 ms 64.656 ms ggr6.la2ca.ip.att.net (12.122.90.61) [AS7018] 66.573 ms cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 65.703 ms 65.694 ms
18 ggr6.la2ca.ip.att.net (12.122.90.61) [AS7018] 66.560 ms 66.549 ms 66.654 ms 66.675 ms 66.545 ms
19 12.81.131.9 (12.81.131.9) [AS7018] 66.521 ms * * * *
20 12.81.131.9 (12.81.131.9) [AS7018] 66.727 ms * * * *
21 * * * * *
22 * * * * *
23 * * * * *
Traceroute from AWS AS14618 in United States, Charlottesville (us-east-2)
traceroute to 107.152.25.197 (107.152.25.197), 50 hops max, 60 byte packets
1 ec2-52-15-0-0.us-east-2.compute.amazonaws.com (52.15.0.0) [AS16509] 1041.666 ms ec2-52-15-0-14.us-east-2.compute.amazonaws.com (52.15.0.14) [AS16509] 20.554 ms ec2-52-15-0-4.us-east-2.compute.amazonaws.com (52.15.0.4) [AS16509] 28.261 ms ec2-52-15-0-6.us-east-2.compute.amazonaws.com (52.15.0.6) [AS16509] 20.534 ms ec2-52-15-0-14.us-east-2.compute.amazonaws.com (52.15.0.14) [AS16509] 20.555 ms
2 100.64.0.136 (100.64.0.136) [AS22773] 16.206 ms 100.64.1.12 (100.64.1.12) [AS22773] 19.959 ms 100.64.2.8 (100.64.2.8) [AS22773] 14.398 ms 100.64.1.202 (100.64.1.202) [AS22773] 19.932 ms 100.64.2.78 (100.64.2.78) [AS22773] 22.115 ms
3 100.64.0.135 (100.64.0.135) [AS22773] 20.634 ms 100.64.2.131 (100.64.2.131) [AS22773] 20.220 ms 100.64.3.3 (100.64.3.3) [AS22773] 20.217 ms 100.64.1.199 (100.64.1.199) [AS22773] 20.657 ms 100.64.0.133 (100.64.0.133) [AS22773] 15.440 ms
4 100.64.17.193 (100.64.17.193) [AS22773] 0.569 ms 0.285 ms 100.64.17.175 (100.64.17.175) [AS22773] 0.291 ms 100.64.17.69 (100.64.17.69) [AS22773] 0.285 ms 100.64.17.197 (100.64.17.197) [AS22773] 0.272 ms
5 52.95.3.131 (52.95.3.131) [AS16509] 1.045 ms 1.051 ms 52.95.1.1 (52.95.1.1) [AS16509] 1.038 ms 52.95.3.131 (52.95.3.131) [AS16509] 1.052 ms 52.95.1.5 (52.95.1.5) [AS16509] 1.046 ms
6 52.95.1.222 (52.95.1.222) [AS16509] 7.259 ms 52.95.1.138 (52.95.1.138) [AS16509] 11.550 ms 11.558 ms 52.95.1.222 (52.95.1.222) [AS16509] 2.018 ms 52.95.1.194 (52.95.1.194) [AS16509] 4.669 ms
7 52.95.1.123 (52.95.1.123) [AS16509] 1.132 ms 52.95.1.91 (52.95.1.91) [AS16509] 6.236 ms 52.95.1.229 (52.95.1.229) [AS16509] 2.090 ms 52.95.1.231 (52.95.1.231) [AS16509] 5.015 ms 52.95.1.235 (52.95.1.235) [AS16509] 1.096 ms
8 54.239.43.162 (54.239.43.162) [AS16509] 12.872 ms 54.239.43.156 (54.239.43.156) [AS16509] 11.780 ms 54.239.43.158 (54.239.43.158) [AS16509] 13.994 ms 54.239.43.156 (54.239.43.156) [AS16509] 11.859 ms 54.239.42.133 (54.239.42.133) [AS16509] 11.559 ms
9 54.239.108.252 (54.239.108.252) [AS16509] 19.438 ms 54.239.108.184 (54.239.108.184) [AS16509] 12.688 ms * 54.239.109.30 (54.239.109.30) [AS16509] 36.114 ms 36.071 ms
10 54.239.108.84 (54.239.108.84) [AS16509] 14.413 ms 54.239.111.243 (54.239.111.243) [AS16509] 11.468 ms 54.239.109.21 (54.239.109.21) [AS16509] 11.560 ms 54.239.108.116 (54.239.108.116) [AS16509] 16.102 ms 54.239.108.105 (54.239.108.105) [AS16509] 11.470 ms
11 206.121.24.33 (206.121.24.33) [*] 12.643 ms 12.247.95.9 (12.247.95.9) [AS7018] 12.286 ms 206.121.24.41 (206.121.24.41) [*] 12.415 ms 206.121.24.33 (206.121.24.33) [*] 12.610 ms 54.239.111.241 (54.239.111.241) [AS16509] 11.603 ms
12 12.122.135.214 (12.122.135.214) [AS7018] 75.296 ms 74.841 ms 206.121.24.33 (206.121.24.33) [*] 12.615 ms 12.122.135.226 (12.122.135.226) [AS7018] 73.702 ms 206.121.24.33 (206.121.24.33) [*] 13.840 ms
13 12.122.135.226 (12.122.135.226) [AS7018] 72.375 ms 12.122.135.214 (12.122.135.214) [AS7018] 72.802 ms cr2.wswdc.ip.att.net (12.122.134.94) [AS7018] 79.258 ms 12.122.135.214 (12.122.135.214) [AS7018] 73.889 ms cr2.wswdc.ip.att.net (12.122.134.94) [AS7018] 79.021 ms
14 cr1.cgcil.ip.att.net (12.122.18.21) [AS7018] 74.908 ms 74.362 ms 73.492 ms cr2.wswdc.ip.att.net (12.122.134.94) [AS7018] 75.664 ms cr1.cgcil.ip.att.net (12.122.18.21) [AS7018] 73.339 ms
15 cr2.dvmco.ip.att.net (12.122.28.77) [AS7018] 77.858 ms 74.313 ms 72.827 ms cr1.cgcil.ip.att.net (12.122.18.21) [AS7018] 75.441 ms 74.925 ms
16 cr1.slkut.ip.att.net (12.122.30.25) [AS7018] 72.827 ms 72.644 ms 75.316 ms 75.297 ms cr2.dvmco.ip.att.net (12.122.28.77) [AS7018] 76.338 ms
17 cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 73.474 ms 73.458 ms cr1.slkut.ip.att.net (12.122.30.25) [AS7018] 74.073 ms 75.833 ms 75.221 ms
18 ggr6.la2ca.ip.att.net (12.122.90.61) [AS7018] 71.910 ms 71.997 ms cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 73.554 ms ggr6.la2ca.ip.att.net (12.122.90.61) [AS7018] 71.980 ms cr1.la2ca.ip.att.net (12.122.1.185) [AS7018] 73.313 ms
19 * * * ggr6.la2ca.ip.att.net (12.122.90.61) [AS7018] 71.809 ms *
20 12.81.131.9 (12.81.131.9) [AS7018] 71.924 ms 71.968 ms 71.923 ms 71.929 ms 72.175 ms
21 * * * * 12.81.131.9 (12.81.131.9) [AS7018] 71.957 ms
22 * * * * 12.81.131.9 (12.81.131.9) [AS7018] 72.029 ms
23 12.81.131.9 (12.81.131.9) [AS7018] 72.192 ms * * * *
24 * 12.81.131.9 (12.81.131.9) [AS7018] 72.214 ms 72.196 ms 72.198 ms *
25 12.81.131.9 (12.81.131.9) [AS7018] 72.199 ms 72.194 ms * 72.196 ms *
26 * * 12.81.131.9 (12.81.131.9) [AS7018] 72.390 ms 72.396 ms 72.173 ms
27 * * * 12.81.131.9 (12.81.131.9) [AS7018] 72.065 ms *
28 * 12.81.131.9 (12.81.131.9) [AS7018] 72.078 ms * 72.005 ms *
29 12.81.131.9 (12.81.131.9) [AS7018] 72.060 ms * * * 72.430 ms
30 12.81.131.9 (12.81.131.9) [AS7018] 72.120 ms * 72.104 ms * 72.129 ms
31 * * * * *
32 * 12.81.131.9 (12.81.131.9) [AS7018] 72.345 ms 72.349 ms 72.374 ms 72.116 ms
33 * * 12.81.131.9 (12.81.131.9) [AS7018] 72.041 ms * *
34 12.81.131.9 (12.81.131.9) [AS7018] 72.367 ms * 72.357 ms * 72.123 ms
35 12.81.131.9 (12.81.131.9) [AS7018] 92.187 ms 72.132 ms * * *
36 * * * * *
37 * * * * *
38 * * * * 12.81.131.9 (12.81.131.9) [AS7018] 72.185 ms
39 * 12.81.131.9 (12.81.131.9) [AS7018] 72.196 ms 72.206 ms 72.215 ms *
40 12.81.131.9 (12.81.131.9) [AS7018] 72.513 ms 72.511 ms * 72.241 ms *
41 12.81.131.9 (12.81.131.9) [AS7018] 72.343 ms * * 72.536 ms 72.328 ms
42 * 12.81.131.9 (12.81.131.9) [AS7018] 72.239 ms 72.475 ms 72.409 ms 72.230 ms
43 * 12.81.131.9 (12.81.131.9) [AS7018] 72.271 ms 72.460 ms 72.322 ms *
44 * 12.81.131.9 (12.81.131.9) [AS7018] 72.278 ms 72.255 ms 72.202 ms *
45 * * 12.81.131.9 (12.81.131.9) [AS7018] 72.222 ms * 72.227 ms
46 12.81.131.9 (12.81.131.9) [AS7018] 72.446 ms * 72.465 ms 72.472 ms *
47 12.81.131.9 (12.81.131.9) [AS7018] 77.258 ms * * 77.387 ms 77.405 ms
48 12.81.131.9 (12.81.131.9) [AS7018] 77.230 ms 77.241 ms 77.228 ms 77.247 ms 77.253 ms
49 * 12.81.131.9 (12.81.131.9) [AS7018] 77.268 ms 77.382 ms 77.283 ms *
50 12.81.131.9 (12.81.131.9) [AS7018] 77.379 ms * 72.359 ms 72.363 ms 72.368 ms
Traceroute from AWS AS16509 in Canada, Montreal (ca-central-1)
traceroute to 107.152.25.197 (107.152.25.197), 50 hops max, 60 byte packets
1 ec2-52-60-0-66.ca-central-1.compute.amazonaws.com (52.60.0.66) [AS16509] 14.171 ms ec2-52-60-0-70.ca-central-1.compute.amazonaws.com (52.60.0.70) [AS16509] 13.661 ms ec2-52-60-0-66.ca-central-1.compute.amazonaws.com (52.60.0.66) [AS16509] 14.164 ms ec2-52-60-0-68.ca-central-1.compute.amazonaws.com (52.60.0.68) [AS16509] 17.033 ms 17.032 ms
2 100.66.0.128 (100.66.0.128) [*] 13.958 ms 100.66.0.212 (100.66.0.212) [*] 13.373 ms 100.66.0.142 (100.66.0.142) [*] 22.104 ms 100.66.0.240 (100.66.0.240) [*] 13.945 ms 100.66.0.206 (100.66.0.206) [*] 22.097 ms
3 100.66.0.55 (100.66.0.55) [*] 20.214 ms 100.66.0.85 (100.66.0.85) [*] 4.222 ms 100.66.0.61 (100.66.0.61) [*] 20.205 ms 100.66.0.69 (100.66.0.69) [*] 4.220 ms 100.66.0.83 (100.66.0.83) [*] 4.206 ms
4 100.65.0.33 (100.65.0.33) [*] 0.336 ms 100.65.0.97 (100.65.0.97) [*] 0.246 ms 100.65.1.129 (100.65.1.129) [*] 0.293 ms 100.65.0.193 (100.65.0.193) [*] 0.265 ms 100.65.0.225 (100.65.0.225) [*] 1.371 ms
5 52.94.81.47 (52.94.81.47) [AS16509] 1.753 ms 1.047 ms 2.704 ms 52.94.81.39 (52.94.81.39) [AS16509] 1.073 ms 52.94.81.51 (52.94.81.51) [AS16509] 3.365 ms
6 52.94.81.226 (52.94.81.226) [AS16509] 7.508 ms 52.94.81.248 (52.94.81.248) [AS16509] 3.884 ms 52.94.81.204 (52.94.81.204) [AS16509] 11.331 ms 52.94.81.226 (52.94.81.226) [AS16509] 4.966 ms 52.94.81.138 (52.94.81.138) [AS16509] 6.595 ms
7 52.94.81.255 (52.94.81.255) [AS16509] 1.509 ms 52.94.81.233 (52.94.81.233) [AS16509] 1.552 ms 52.94.81.189 (52.94.81.189) [AS16509] 1.169 ms 2.167 ms 1.883 ms
8 54.239.44.18 (54.239.44.18) [AS16509] 10.804 ms 10.864 ms 10.862 ms 10.861 ms 10.865 ms
9 52.93.4.27 (52.93.4.27) [AS16509] 10.856 ms 52.93.4.17 (52.93.4.17) [AS16509] 10.887 ms 52.93.4.19 (52.93.4.19) [AS16509] 15.166 ms 52.93.4.17 (52.93.4.17) [AS16509] 10.896 ms 52.93.4.19 (52.93.4.19) [AS16509] 15.162 ms
10 52.93.4.94 (52.93.4.94) [AS16509] 10.834 ms 52.93.4.96 (52.93.4.96) [AS16509] 10.878 ms 52.93.4.106 (52.93.4.106) [AS16509] 10.874 ms 52.93.4.90 (52.93.4.90) [AS16509] 10.890 ms 52.93.4.86 (52.93.4.86) [AS16509] 10.864 ms
11 12.247.222.9 (12.247.222.9) [AS7018] 11.545 ms 206.121.240.25 (206.121.240.25) [*] 11.467 ms 12.247.222.9 (12.247.222.9) [AS7018] 11.697 ms 206.121.240.25 (206.121.240.25) [*] 11.494 ms 12.247.222.9 (12.247.222.9) [AS7018] 12.594 ms
12 12.122.115.62 (12.122.115.62) [AS7018] 77.791 ms 12.122.115.34 (12.122.115.34) [AS7018] 73.001 ms 73.025 ms 75.865 ms 12.122.115.62 (12.122.115.62) [AS7018] 76.076 ms
13 cr2.cgcil.ip.att.net (12.122.2.237) [AS7018] 74.508 ms 74.483 ms 74.489 ms 74.495 ms 74.484 ms
14 cr1.cgcil.ip.att.net (12.122.2.53) [AS7018] 75.777 ms 75.047 ms 75.000 ms 75.002 ms 74.998 ms
15 cr2.dvmco.ip.att.net (12.122.28.77) [AS7018] 74.239 ms 78.222 ms 74.231 ms 78.224 ms 74.825 ms
16 cr1.slkut.ip.att.net (12.122.30.25) [AS7018] 78.135 ms 78.141 ms 78.130 ms 78.127 ms 78.128 ms
17 cr1.la2ca.ip.att.net (12.122.30.30) [AS7018] 76.652 ms 76.642 ms 76.644 ms 76.646 ms 76.643 ms
18 ggr6.la2ca.ip.att.net (12.122.90.61) [AS7018] 73.914 ms 73.828 ms 73.849 ms 73.806 ms 73.869 ms
19 * * * * *
20 12.81.131.9 (12.81.131.9) [AS7018] 74.060 ms 73.965 ms 73.993 ms 74.066 ms 74.009 ms
21 * * * * *
22 12.81.131.9 (12.81.131.9) [AS7018] 74.028 ms * * * *
23 * * * * *
24 * * * * *
25 * * * * *
On 2017-08-19 20:44, Sahil Ganguly via Outages wrote:
> Is anyone from AT&T on this mailing list that could help us out in pushing for a fix? We're seeing some issues that look like a routing loop getting from AT&T networks:
>
> $ traceroute account.box.com <http://account.box.com>
> traceroute: Warning: account.box.com <http://account.box.com> has multiple addresses; using 107.152.25.197
> traceroute to account.box.com <http://account.box.com> (107.152.25.197), 64 hops max, 52 byte packets
> 1 172.20.10.1 (172.20.10.1) 23.643 ms 16.233 ms 4.400 ms
> 2 172.26.96.169 (172.26.96.169) 88.196 ms 49.009 ms 21.336 ms
> 3 172.16.126.36 (172.16.126.36) 31.343 ms
> 172.16.126.60 (172.16.126.60) 530.635 ms 201.699 ms
> 4 12.249.2.49 (12.249.2.49) 39.212 ms 104.090 ms 32.640 ms
> 5 12.83.180.82 (12.83.180.82) 49.394 ms 50.699 ms 40.146 ms
> 6 cr1.sl9mo.ip.att.net <http://cr1.sl9mo.ip.att.net> (12.122.18.9) 41.180 ms 39.546 ms 39.920 ms
> 7 ggr6.la2ca.ip.att.net <http://ggr6.la2ca.ip.att.net> (12.122.90.61) 36.784 ms 37.823 ms 41.714 ms
> 8 * * *
> 9 12.81.131.9 (12.81.131.9) 57.906 ms 66.188 ms 44.169 ms
> 10 * * *
> 11 12.81.131.9 (12.81.131.9) 686.796 ms 163.466 ms 43.789 ms
> 12 * * *
> 13 * 12.81.131.9 (12.81.131.9) 160.871 ms 54.204 ms
> 14 * * *
> 15 12.81.131.9 (12.81.131.9) 85.982 ms * *
> 16 *
>
>
> Thanks!
>
> --
> Sahil Ganguly
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
More information about the Outages-discussion
mailing list