[outages] IPv6 tunnels in FRA1 on HE.net down?

[Constantine A. Murenin]

I'm being told by the NOC that someone is on the ground in FRA1,
replacing a switch; and that it's a "scheduled maintenance", no ETA,
and there was no announcement, since someone has to take a flight to
Europe and its unpredictable. :-)

35 minutes of downtime, and counting...

C.

On 12 June 2013 15:19, Constantine A. Murenin <mureninc at gmail.com> wrote:
> I smell a big outage:
>
> % date; traceroute tserv1.fra1.he.net
> Wed Jun 12 15:17:04 PDT 2013
> traceroute to tserv1.fra1.he.net (216.66.80.30), 30 hops max, 60 byte packets
>  1  192.168.105.3 (192.168.105.3)  0.673 ms  0.773 ms  0.923 ms
>  2  10gigabitethernet7-6.core3.fmt2.he.net (65.49.10.217)  1.795 ms
> 1.811 ms  1.795 ms
>  3  10gigabitethernet12-1.core1.lax1.he.net (184.105.213.26)  19.773
> ms 10gigabitethernet10-1.core1.sjc2.he.net (184.105.222.14)  0.786 ms
> 0.756 ms
>  4  10gigabitethernet10-8.core1.nyc4.he.net (72.52.92.225)  71.714 ms
> 76.569 ms 10gigabitethernet14-2.core1.nyc4.he.net (184.105.213.198)
> 71.690 ms
>  5  * * *
>  6  * * *
>  7  * * *
>  8  * * *
>  9  * * *
> 10  * * *
> 11  * * *
> 12  * * *
> 13  * * *
> 14  * * *
> 15  * *^C
>
> However, a "reverse" traceroute works fine:
>
> % date; traceroute ns1.he.net
> Wed Jun 12 15:18:47 PDT 2013
> traceroute to ns1.he.net (216.218.130.2), 64 hops max, 40 byte packets
>  1  static.33.203.4.46.clients.your-server.de (46.4.203.33)  0.682 ms
> 0.531 ms  0.481 ms
>  2  hos-tr4.juniper2.rz13.hetzner.de (213.239.224.97)  0.245 ms
> hos-tr3.juniper2.rz13.hetzner.de (213.239.224.65)  0.242 ms
> hos-tr2.juniper1.rz13.hetzner.de (213.239.224.33)  0.241 ms
>  3  hos-bb2.juniper4.ffm.hetzner.de (213.239.240.150)  5.951 ms
> hos-bb1.juniper1.ffm.hetzner.de (213.239.240.224)  4.803 ms  4.786 ms
>  4  30gigabitethernet4-3.core1.fra1.he.net (80.81.192.172)  6.354 ms
> 6.81 ms  5.451 ms
>  5  10gigabitethernet10-2.core1.par2.he.net (72.52.92.26)  22.803 ms
> 24.531 ms  24.787 ms
>  6  10gigabitethernet15-1.core1.ash1.he.net (184.105.213.93)  101.504
> ms  99.563 ms  99.958 ms
>  7  10gigabitethernet11-1.core1.pao1.he.net (184.105.213.177)  163.687
> ms  171.711 ms  175.34 ms
>  8  10gigabitethernet1-2.core1.fmt1.he.net (184.105.213.65)  163.940
> ms  171.362 ms  167.67 ms
>  9  ns1.he.net (216.218.130.2)  163.52 ms  165.265 ms  164.143 ms
>
> C.
>
> On 12 June 2013 15:11, Constantine A. Murenin <mureninc at gmail.com> wrote:
>> he.net fra1 tserv1 is down since about 10 minutes ago (~14:57 PT),
>> this time it seems like the whole thing is down, cannot even ping
>> ordns.he.net over IPv6; a connection of my friend who's running a
>> smokeping is also down, e.g. this is definitely widespread.
>>
>> Not sure what's exactly down, but it seems to be bgp-related, perhaps:
>>
>>  1  2600:3c01::8678:acff:fe0d:79c1 (2600:3c01::8678:acff:fe0d:79c1)
>> 0.699 ms  0.828 ms  0.970 ms
>>  2  10gigabitethernet7-6.core3.fmt2.he.net (2001:470:1:3b8::1)  5.936
>> ms  5.877 ms  5.859 ms
>>  3  10gigabitethernet5-4.core1.pao1.he.net (2001:470:0:263::2)  6.743
>> ms  6.877 ms  6.978 ms
>>  4  * * *
>>  5  * * *
>>  6  * * *
>>  7  * * *
>>  8  * * *
>>  9  * * *
>> 10  * * *
>> 11  * * *
>> 12  * * *
>> 13  * * *
>> 14  * * *
>> 15  * * *
>> 16  * * *
>> 17  * * *
>> 18  * * *
>>
>> C.