I just heard back from Level3 confirming that there is an issue affecting multiple customers being investigated, although I don't have any details. My last test showed that it would work if I turned it back up, but it would route from STL to Denver and then to CHI:<div>
<br></div><div><div>Tracing the route to <a href="http://tcsmonitor.trendcs.com">tcsmonitor.trendcs.com</a> (198.17.225.17)</div><div><br></div><div> 1 <a href="http://vlan372.car2.StLouis1.Level3.net">vlan372.car2.StLouis1.Level3.net</a> (4.53.162.241) [AS 3356] 80 msec 216 msec 132 msec</div>
<div> 2 <a href="http://ae-11-11.car1.StLouis1.Level3.net">ae-11-11.car1.StLouis1.Level3.net</a> (4.69.132.185) [AS 3356] [MPLS: Label 325 Exp 0] 244 msec 204 msec 116 msec</div><div> 3 <a href="http://ae-4-4.ebr1.Denver1.Level3.net">ae-4-4.ebr1.Denver1.Level3.net</a> (4.69.132.182) [AS 3356] [MPLS: Label 1481 Exp 0] 20 msec 32 msec 32 msec</div>
<div> 4 <a href="http://ae-3-3.ebr1.Chicago2.Level3.net">ae-3-3.ebr1.Chicago2.Level3.net</a> (4.69.132.62) [AS 3356] [MPLS: Label 1173 Exp 0] 52 msec 48 msec 48 msec</div><div> 5 <a href="http://ae-6-6.ebr1.Chicago1.Level3.net">ae-6-6.ebr1.Chicago1.Level3.net</a> (4.69.140.189) [AS 3356] [MPLS: Label 1096 Exp 0] 56 msec</div>
<div> <a href="http://ae-1-100.ebr2.Chicago2.Level3.net">ae-1-100.ebr2.Chicago2.Level3.net</a> (4.69.132.114) [AS 3356] [MPLS: Label 1233 Exp 0] 48 msec</div><div> <a href="http://ae-6-6.ebr1.Chicago1.Level3.net">ae-6-6.ebr1.Chicago1.Level3.net</a> (4.69.140.189) [AS 3356] [MPLS: Label 1096 Exp 0] 56 msec</div>
<div> 6 * * * </div><div> 7 <a href="http://ae-1-9.bar2.Cleveland1.Level3.net">ae-1-9.bar2.Cleveland1.Level3.net</a> (4.69.136.193) [AS 3356] [MPLS: Label 299776 Exp 0] 52 msec 52 msec 52 msec</div><div> 8 <a href="http://ae-0-11.bar1.Cleveland1.Level3.net">ae-0-11.bar1.Cleveland1.Level3.net</a> (4.69.136.185) [AS 3356] 56 msec 56 msec 56 msec</div>
<div> 9 <a href="http://FIDELITY-AC.bar1.Cleveland1.Level3.net">FIDELITY-AC.bar1.Cleveland1.Level3.net</a> (4.53.196.30) [AS 3356] 56 msec 56 msec 56 msec</div><div> 10 <a href="http://ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net">ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net</a> (66.94.64.7) [AS 22958] 56 msec 56 msec 56 msec</div>
<div> 11 <a href="http://net-207-58-250-91.arpa.fidelityaccess.net">net-207-58-250-91.arpa.fidelityaccess.net</a> (207.58.250.91) [AS 22958] 52 msec 56 msec 56 msec</div></div><div><br></div><div><br></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Thu, Dec 13, 2012 at 9:10 AM, Mitch <span dir="ltr"><<a href="mailto:mitpatterson@gmail.com" target="_blank">mitpatterson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Not sure what your issue is. I'm in Ohio so from here I don't hit chicago but I did traceroutes from Level3's looking glass in both Chicago and St. Louis(since it looks likes thats where you are coming from:<div>
<br></div><div>Chicago makes it to the last hop before the last host, not sure if the last host blocks it:</div><div><br></div><div><font color="#0000ff" face="Helvetica, Arial, Sans"><b>Show Level 3 (Chicago, IL) Traceroute to 198.17.225.17</b>
</font><p>
</p><pre><font color="#000000" face="terminal" size="1"> 1 <a href="http://ae-32-52.ebr2.Chicago1.Level3.net" target="_blank">ae-32-52.ebr2.Chicago1.Level3.net</a> (4.69.138.62) 0 msec 0 msec 0 msec
2 <a href="http://ae-1-9.bar2.Cleveland1.Level3.net" target="_blank">ae-1-9.bar2.Cleveland1.Level3.net</a> (4.69.136.193) 8 msec 12 msec 8 msec
3 <a href="http://ae-0-11.bar1.Cleveland1.Level3.net" target="_blank">ae-0-11.bar1.Cleveland1.Level3.net</a> (4.69.136.185) 12 msec 8 msec 8 msec
4 <a href="http://FIDELITY-AC.bar1.Cleveland1.Level3.net" target="_blank">FIDELITY-AC.bar1.Cleveland1.Level3.net</a> (4.53.196.30) 12 msec 8 msec 12 msec
5 <a href="http://ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net" target="_blank">ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net</a> (66.94.64.7) [AS22958 {FIDELITY-001}] 8 msec 8 msec 12 msec
6 <a href="http://net-207-58-250-91.arpa.fidelityaccess.net" target="_blank">net-207-58-250-91.arpa.fidelityaccess.net</a> <a href="tel:%28207.58.250.91" value="+12075825091" target="_blank">(207.58.250.91</a>) [AS22958 {FIDELITY-001}] 8 msec 8 msec 12 msec
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * * timeout !
</font></pre>From St. Louis it shows the same thing. It definity makes it though chicago:</div><div><font face="Helvetica, Arial, Sans" color="#0000ff"><b>Show Level 3 (Saint Louis, MO) Traceroute to 198.17.225.17</b></font><p style="font-family:'Times New Roman';font-size:medium">
</p><pre><font face="terminal" size="1" color="#000000"> 1 <a href="http://ae-11-11.car2.StLouis1.Level3.net" target="_blank">ae-11-11.car2.StLouis1.Level3.net</a> (4.69.132.186) 192 msec 268 msec 208 msec
2 <a href="tel:4.69.201.1%20100" value="+14692011100" target="_blank">4.69.201.1 100</a> msec 84 msec 68 msec
3 <a href="http://ae-6-6.ebr1.Chicago1.Level3.net" target="_blank">ae-6-6.ebr1.Chicago1.Level3.net</a> (4.69.140.189) 40 msec
<a href="http://ae-1-100.ebr2.Chicago2.Level3.net" target="_blank">ae-1-100.ebr2.Chicago2.Level3.net</a> (4.69.132.114) 148 msec 196 msec
4 <a href="http://ae-5-5.ebr2.Chicago1.Level3.net" target="_blank">ae-5-5.ebr2.Chicago1.Level3.net</a> (4.69.140.193) 8 msec
<a href="http://ae-1-100.ebr2.Chicago1.Level3.net" target="_blank">ae-1-100.ebr2.Chicago1.Level3.net</a> (4.69.151.178) 4 msec 12 msec
5 <a href="http://ae-1-9.bar2.Cleveland1.Level3.net" target="_blank">ae-1-9.bar2.Cleveland1.Level3.net</a> (4.69.136.193) 36 msec 16 msec 16 msec
6 <a href="http://ae-0-11.bar1.Cleveland1.Level3.net" target="_blank">ae-0-11.bar1.Cleveland1.Level3.net</a> (4.69.136.185) 12 msec 16 msec 12 msec
7 <a href="http://FIDELITY-AC.bar1.Cleveland1.Level3.net" target="_blank">FIDELITY-AC.bar1.Cleveland1.Level3.net</a> (4.53.196.30) 16 msec 12 msec 16 msec
8 <a href="http://ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net" target="_blank">ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net</a> (66.94.64.7) [AS22958 {FIDELITY-001}] 16 msec 16 msec 16 msec
9 <a href="http://net-207-58-250-91.arpa.fidelityaccess.net" target="_blank">net-207-58-250-91.arpa.fidelityaccess.net</a> <a href="tel:%28207.58.250.91" value="+12075825091" target="_blank">(207.58.250.91</a>) [AS22958 {FIDELITY-001}] 16 msec 16 msec 16 msec
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * * timeout !</font></pre><div class="gmail_quote"><div class="im">On Thu, Dec 13, 2012 at 9:50 AM, Jason Baugher <span dir="ltr"><<a href="mailto:jason@thebaughers.com" target="_blank">jason@thebaughers.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">Starting at 8am Central Time, we're seeing issues with Level3 in Chicago. I don't have access to the far end to do a traceroute from the far side.<div>
<br></div><div>Traceroute with our Level3 peering up:</div><div>
<br></div><div><div>Tracing the route to <a href="http://tcsmonitor.trendcs.com" target="_blank">tcsmonitor.trendcs.com</a> (198.17.225.17)</div><div><br></div><div> 1 <a href="http://vlan372.car2.StLouis1.Level3.net" target="_blank">vlan372.car2.StLouis1.Level3.net</a> (4.53.162.241) [AS 3356] 160 msec 8 msec 4 msec</div>
<div> 2 4.69.201.1 [AS 3356] [MPLS: Label 1173 Exp 0] 16 msec 8 msec 16 msec</div><div> 3 <a href="http://ae-1-100.ebr2.Chicago2.Level3.net" target="_blank">ae-1-100.ebr2.Chicago2.Level3.net</a> (4.69.132.114) [AS 3356] [MPLS: Label 1233 Exp 0] 8 msec</div>
<div> <a href="http://ae-6-6.ebr1.Chicago1.Level3.net" target="_blank">ae-6-6.ebr1.Chicago1.Level3.net</a> (4.69.140.189) [AS 3356] [MPLS: Label 1096 Exp 0] 12 msec</div><div> <a href="http://ae-1-100.ebr2.Chicago2.Level3.net" target="_blank">ae-1-100.ebr2.Chicago2.Level3.net</a> (4.69.132.114) [AS 3356] [MPLS: Label 1233 Exp 0] 8 msec</div>
<div> 4 * * * </div><div> 5 * * * </div><div> 6 * * * </div></div><div><br></div><div><br></div><div>Traceroute with it down, traffic moved to Sprint:</div><div><br></div><div><div>Tracing the route to <a href="http://tcsmonitor.trendcs.com" target="_blank">tcsmonitor.trendcs.com</a> (198.17.225.17)</div>
<div><br></div><div> 1 <a href="http://qncy-pc-cat-0001.adams.net" target="_blank">qncy-pc-cat-0001.adams.net</a> (66.226.125.198) 4 msec 0 msec 0 msec</div><div> 2 <a href="http://sl-st20-chi-9-0-1.sprintlink.net" target="_blank">sl-st20-chi-9-0-1.sprintlink.net</a> (144.223.60.49) [AS 1239] 8 msec 4 msec 8 msec</div>
<div> 3 144.232.8.114 [AS 1239] 8 msec 4 msec 4 msec</div><div> 4 <a href="http://vlan52.ebr2.Chicago2.Level3.net" target="_blank">vlan52.ebr2.Chicago2.Level3.net</a> (4.69.138.190) [AS 3356] [MPLS: Label 1233 Exp 0] 8 msec 16 msec 4 msec</div>
<div> 5 * * * </div><div> 6 <a href="http://ae-1-9.bar2.Cleveland1.Level3.net" target="_blank">ae-1-9.bar2.Cleveland1.Level3.net</a> (4.69.136.193) [AS 3356] [MPLS: Label 299776 Exp 0] 24 msec 64 msec 52 msec</div><div>
7 <a href="http://ae-0-11.bar1.Cleveland1.Level3.net" target="_blank">ae-0-11.bar1.Cleveland1.Level3.net</a> (4.69.136.185) [AS 3356] 24 msec 24 msec 24 msec</div>
<div> 8 <a href="http://FIDELITY-AC.bar1.Cleveland1.Level3.net" target="_blank">FIDELITY-AC.bar1.Cleveland1.Level3.net</a> (4.53.196.30) [AS 3356] 136 msec 212 msec 212 msec</div><div> 9 <a href="http://ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net" target="_blank">ge-1-0-1.ar2.kth.cle.oh.fidelityaccess.net</a> (66.94.64.7) [AS 22958] 16 msec 16 msec 16 msec</div>
<div> 10 <a href="http://net-207-58-250-91.arpa.fidelityaccess.net" target="_blank">net-207-58-250-91.arpa.fidelityaccess.net</a> <a href="tel:%28207.58.250.91" value="+12075825091" target="_blank">(207.58.250.91</a>) [AS 22958] 16 msec 20 msec 16 msec</div>
</div><div><br></div><div><br></div><div>We've opened a ticket with Level3, just wondering if anyone else has noticed problems this morning.</div>
<br></div></div><div class="im">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div><br></div>