[outages] HTTP was timing out with high latency to v6 version of www.charter.com and ipv6.cnn.com

Wed Oct 19 12:44:22 EDT 2011

Started at 10:45 am Central, cleared up by 11:17 am Central.

==============
Before
==============
nagios:/var/log# tcptraceroute6 www.charter.com
traceroute to www.charter.com (2607:f428:3:1:80:80:80:1) from
2607:fe28:0:1000::5, port 80, from port 47336, 30 hops max, 60 bytes packets
 1  2607:fe28:0:1003::2 (2607:fe28:0:1003::2)  1.070 ms  1.490 ms  1.392 ms
 2  router-core.mtcnet.net (2607:fe28:0:1000::1)  1.944 ms  1.768 ms  1.661
ms
 3  sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197)  2.090 ms  1.906 ms  1.350
ms
 4  v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5)  3.759 ms  2.979
ms  2.658 ms
 5  v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1)  7.358 ms
6.969 ms  6.715 ms
 6  v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1)  9.249 ms  8.823 ms
8.477 ms
 7  2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1)  21.343 ms  20.996
ms  20.693 ms
 8  2001:428::205:171:202:195 (2001:428::205:171:202:195)  33.249 ms  32.945
ms  32.716 ms
 9  * * *
10  * * *
11  2607:f428:3:1:80:80:80:1 (2607:f428:3:1:80:80:80:1)  3184.745 ms [open]
3184.899 ms  183.671 ms
nagios:/var/log#

==============
After
==============
nagios:/var/log# tcptraceroute6 www.charter.com
traceroute to www.charter.com (2607:f428:3:1:80:80:80:1) from
2607:fe28:0:1000::5, port 80, from port 37668, 30 hops max, 60 bytes packets
 1  2607:fe28:0:1003::2 (2607:fe28:0:1003::2)  1.734 ms  1.230 ms  1.828 ms
 2  router-core.mtcnet.net (2607:fe28:0:1000::1)  1.610 ms  3.112 ms  6.192
ms
 3  sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197)  6.111 ms  6.498 ms  1.429
ms
 4  v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5)  8.275 ms  3.175
ms  3.589 ms
 5  v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1)  93.316 ms
8.445 ms  6.854 ms
 6  v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1)  10.056 ms  8.804 ms
11.640 ms
 7  2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1)  21.810 ms  24.138
ms  23.784 ms
 8  2001:428::205:171:202:195 (2001:428::205:171:202:195)  36.477 ms  42.724
ms  32.726 ms
 9  * * *
10  * * *
11  2607:f428:3:1:80:80:80:1 (2607:f428:3:1:80:80:80:1)  69.142 ms [open]
68.620 ms  68.139 ms
nagios:/var/log#

==============
Before
==============
nagios:/var/log# tcptraceroute6 ipv6.cnn.com
traceroute to ipv6.cnn.com (2620:100:e000::8001) from 2607:fe28:0:1000::5,
port 80, from port 52863, 30 hops max, 60 bytes packets
 1  2607:fe28:0:1003::2 (2607:fe28:0:1003::2)  1.090 ms  1.209 ms  1.889 ms
 2  router-core.mtcnet.net (2607:fe28:0:1000::1)  1.608 ms  2.262 ms  1.480
ms
 3  sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197)  2.583 ms  2.059 ms  1.232
ms
 4  v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5)  3.871 ms  3.064
ms  2.676 ms
 5  v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1)  7.450 ms
7.005 ms  6.739 ms
 6  v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1)  9.206 ms  8.867 ms
8.480 ms
 7  2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1)  21.453 ms  20.980
ms  20.707 ms
 8  2001:428:0:1:205:171:202:209 (2001:428:0:1:205:171:202:209)  45.218 ms
44.844 ms  44.537 ms
 9  ae5.edge2.dallas3.level3.net (2001:1900:4:3::1c1)  45.079 ms  44.733 ms
44.361 ms
10  * vl-80.car2.Dallas1.Level3.net (2001:1900:17:3::4)  45.890 ms  *
11  vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96)  188.828 ms
188.544 ms  188.134 ms
12  vl-52.car1.Atlanta1.Level3.net (2001:1900:1c:2::5)  188.158 ms  187.907
ms  187.752 ms
13  TIME-WARNER.car1.Atlanta1.Level3.net (2001:1900:2100::496)  189.189 ms
* 188.380 ms
14  2620:100:e000:ffff::21 (2620:100:e000:ffff::21)  76.773 ms  76.348 ms
75.971 ms
15  * 2620:100:e000:ffff::12 (2620:100:e000:ffff::12)  190.214 ms  *
16  * 2620:100:e000:ffff::25 (2620:100:e000:ffff::25)  189.459 ms  189.274
ms
17  * * *
18  * * 2620:100:e000::8001 (2620:100:e000::8001)  189.209 ms [open]
19  2620:100:e000::8001 (2620:100:e000::8001)  189.731 ms [open]  189.373 ms
189.482 ms
nagios:/var/log#

==============
After
==============
nagios:/var/log# tcptraceroute6 ipv6.cnn.com
traceroute to ipv6.cnn.com (2620:100:e000::8001) from 2607:fe28:0:1000::5,
port 80, from port 36895, 30 hops max, 60 bytes packets
 1  2607:fe28:0:1003::2 (2607:fe28:0:1003::2)  1.120 ms  1.622 ms  1.307 ms
 2  router-core.mtcnet.net (2607:fe28:0:1000::1)  1.926 ms  1.698 ms  1.771
ms
 3  sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197)  1.997 ms  2.013 ms  1.736
ms
 4  v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5)  3.830 ms  3.122
ms  2.693 ms
 5  v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1)  7.786 ms
7.108 ms  6.820 ms
 6  v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1)  9.158 ms  8.758 ms
8.498 ms
 7  2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1)  21.350 ms  21.007
ms  38.574 ms
 8  2001:428:0:1:205:171:202:209 (2001:428:0:1:205:171:202:209)  47.081 ms
44.443 ms  44.111 ms
 9  ae5.edge2.dallas3.level3.net (2001:1900:4:3::1c1)  44.579 ms  92.648 ms
43.891 ms
10  vl-60.car2.Dallas1.Level3.net (2001:1900:17:1::4)  157.562 ms  142.664
ms  225.796 ms
11  vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96)  146.571 ms  89.600
ms  127.063 ms
12  vl-52.car1.Atlanta1.Level3.net (2001:1900:1c:2::5)  73.790 ms  73.028 ms
72.797 ms
13  TIME-WARNER.car1.Atlanta1.Level3.net (2001:1900:2100::496)  118.018 ms
73.072 ms  72.854 ms
14  2620:100:e000:ffff::21 (2620:100:e000:ffff::21)  75.514 ms  74.957 ms
74.781 ms
15  2620:100:e000:ffff::12 (2620:100:e000:ffff::12)  74.907 ms  74.363 ms
73.949 ms
16  2620:100:e000:ffff::25 (2620:100:e000:ffff::25)  75.807 ms  74.881 ms
74.565 ms
17  2620:100:e000:ffff::32 (2620:100:e000:ffff::32)  76.823 ms  74.865 ms
74.626 ms
18  2620:100:e000::8001 (2620:100:e000::8001)  74.190 ms [open]  73.974 ms
73.231 ms
nagios:/var/log#