[outages] IPv6 tunnels in FRA1 on HE.net down?

Tue Apr 23 22:28:50 EDT 2013

On 23 April 2013 18:28, Constantine A. Murenin <mureninc at gmail.com> wrote:
> As of a couple of minutes ago, my IPv6 tunnel seems to have no
> connectivity, weirdly other than being able to access ordns.he.net
> (2001:470:20::2) just fine, but not ns{2,3,4,5}.he.net, or any other
> IPv6 host.

Update: reported to he.net support@ on 18:34, including a follow-up phone call; 
everything's back online, as of at least 18:51 PT.  
Pretty fast resolution, for a free service. :-)

According to he.net, tserv wasn't responding on its IPv6 address, and has 
henceforth been rebooted.

Which adds up as per my mtr from a Linode:

# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" -6 XXXXXXXXX ; date
  2. 10gigabitethernet2-3.core1.fmt1.he.net     60    60  0.0%    0.4   2.5   8.5  85.5  15.2
  3. 10gigabitethernet1-2.core1.sjc2.he.net     60    60  0.0%    0.8   2.5   5.3  51.6   8.5
  4. 10gigabitethernet3-3.core1.den1.he.net     60    60  0.0%   27.8  31.7  32.3  70.0   7.3
  5. 10gigabitethernet5-5.core1.mci3.he.net     60    60  0.0%   39.7  43.6  44.3 114.3  10.1
  6. 10gigabitethernet5-2.core1.chi1.he.net     60    60  0.0%   52.0  56.2  57.4 177.2  17.0
  7. 100gigabitethernet7-2.core1.nyc4.he.net    60    59  1.7%   69.1  72.2  72.5 119.8   7.1
  8. 10gigabitethernet1-2.core1.lon1.he.net     60    60  0.0%  137.8 141.8 142.2 229.2  12.0
  9. 10gigabitethernet4-2.core1.fra1.he.net     60    60  0.0%  149.4 154.0 154.4 242.8  12.7
 10. ???                                        60     0 100.0    0.0   0.0   0.0   0.0   0.0
Tue Apr 23 18:14:29 PDT 2013

...

  2. 10gigabitethernet2-3.core1.fmt1.he.net     60    60  0.0%    0.4   2.8   9.3  78.3  16.8
  3. 10gigabitethernet1-2.core1.sjc2.he.net     60    60  0.0%    0.8   2.3   3.9  13.9   3.8
  4. 10gigabitethernet3-3.core1.den1.he.net     60    60  0.0%   27.8  30.3  30.5  39.2   3.5
  5. 10gigabitethernet5-5.core1.mci3.he.net     60    60  0.0%   39.7  43.4  43.6  52.8   4.4
  6. 10gigabitethernet5-2.core1.chi1.he.net     60    60  0.0%   52.0  54.1  54.2  62.2   3.2
  7. 100gigabitethernet7-2.core1.nyc4.he.net    60    60  0.0%   69.1  71.5  71.6  82.1   3.5
  8. 10gigabitethernet1-2.core1.lon1.he.net     60    60  0.0%  137.8 140.3 140.4 149.1   3.5
  9. 10gigabitethernet4-2.core1.fra1.he.net     60    60  0.0%  149.4 152.3 152.3 162.5   3.7
 10. tserv1.fra1.he.net                         60    60  0.0%  154.1 155.5 155.5 163.4   2.2
 11. IPv6.XXXXXX                                60    59  1.7%  155.2 156.0 156.1 162.9   1.2
Tue Apr 23 18:55:34 PDT 2013

And I guess ordns.he.net (2001:470:20::2) really runs on tserv 
(and hence wasn't affected during the outage).

Cns# echo {ordns,ns{2,3,4,5}}.he.net | xargs -n1 traceroute6 -l; date
traceroute6 to ordns.he.net (2001:470:20::2) from 2001:470:XXXX:YYYY::2, 64 hops max, 12 byte packets
 1  ordns.he.net (2001:470:20::2)  6.39 ms  6.495 ms  6.213 ms
traceroute6 to ns2.he.net (2001:470:200::2) from 2001:470:XXXX:YYYY::2, 64 hops max, 12 byte packets
 1  XXXXXX.tunnel.tserv6.fra1.ipv6.he.net (2001:470:XXXX:YYYY::1)  9.353 ms  8.783 ms  9.328 ms
 2  gige-g2-20.core1.fra1.he.net (2001:470:0:69::1)  13.12 ms  15.134 ms  6.252 ms
 3  10gigabitethernet5-3.core1.lon1.he.net (2001:470:0:1d2::1)  18.113 ms  23.554 ms  20.342 ms
 4  ns2.he.net (2001:470:200::2)  20.449 ms  22.873 ms  20.517 ms
traceroute6 to ns3.he.net (2001:470:300::2) from 2001:470:XXXX:YYYY::2, 64 hops max, 12 byte packets
 1  * XXXXXX.tunnel.tserv6.fra1.ipv6.he.net (2001:470:XXXX:YYYY::1)  8.743 ms  9.2 ms
 2  gige-g2-20.core1.fra1.he.net (2001:470:0:69::1)  5.934 ms  10.749 ms  6.197 ms
 3  10gigabitethernet5-3.core1.ams1.he.net (2001:470:0:225::1)  18.567 ms  17.728 ms  13.282 ms
 4  ns3.he.net (2001:470:300::2)  13.462 ms  13.412 ms  13.525 ms
traceroute6 to ns4.he.net (2001:470:400::2) from 2001:470:XXXX:YYYY::2, 64 hops max, 12 byte packets
 1  XXXXXX.tunnel.tserv6.fra1.ipv6.he.net (2001:470:XXXX:YYYY::1)  8.838 ms  8.684 ms  8.682 ms
 2  gige-g2-20.core1.fra1.he.net (2001:470:0:69::1)  6.233 ms  13.208 ms  5.97 ms
 3  ns4.he.net (2001:470:400::2)  6.145 ms  6.38 ms  6.384 ms
traceroute6 to ns5.he.net (2001:470:500::2) from 2001:470:XXXX:YYYY::2, 64 hops max, 12 byte packets
 1  * XXXXXX.tunnel.tserv6.fra1.ipv6.he.net (2001:470:XXXX:YYYY::1)  8.655 ms  8.848 ms
 2  gige-g2-20.core1.fra1.he.net (2001:470:0:69::1)  12.607 ms  6.527 ms  11.929 ms
 3  10gigabitethernet5-3.core1.ams1.he.net (2001:470:0:225::1)  13.289 ms  13.968 ms  16.754 ms
 4  ns5.he.net (2001:470:500::2)  14.111 ms  13.575 ms  13.385 ms
Tue Apr 23 18:59:48 PDT 2013

However, it's unclear why IPv6 connectivity of tserv1.fra1.he.net 
doesn't seem to be monitored otherwise. :/

Cheers,
Constantine.