[outages] Comcast/Level 3 Issues - Detroit/Chicago Areas -- Now CenturyLink today too?

Thu May 11 10:07:14 EDT 2017

Does anyone have any information on some ongoing issues with Comcast in the Michigan area since end of April that involve intermittent packet loss, latency spikes (from normal 20 ms to 500+ ms), and just overall cruddy paths?

We're seeing issues with 3 of our sites in West/Central Michigan that use Comcast Business for Internet as they route from Comcast to Level 3.  Traceroutes seem to die somewhere in Chicago or leaving Detroit either in Level 3 or at the edge of Comcast:

Tracing the route to 172.87.34.1
VRF info: (vrf in name/id, vrf out name/id)
  1 50-77-204-206-static.hfc.comcastbusiness.net (50.77.204.206) 52 msec 40 msec 32 msec
  2 96.120.42.37 32 msec 44 msec 40 msec
  3 xe-7-0-6-sur02.muskegon.mi.michigan.comcast.net (68.85.189.25) 52 msec 52 msec 56 msec
  4 xe-1-3-1-0-sur01.lyndon.mi.michigan.comcast.net (68.87.189.177) 60 msec 68 msec 64 msec
  5 ae2.bar1.detroit1.level3.net (4.68.71.33) 72 msec 64 msec 64 msec
  6  *  *  *
  7 lynx-networ.ear3.chicago2.level3.net (4.35.108.250) 136 msec 144 msec 152 msec
  8 66.35.87.172.in-addr.arpa.lynxnetworkgroup.com (172.87.34.1) 148 msec *  220 msec

Tracing the route to 172.87.34.1
VRF info: (vrf in name/id, vrf out name/id)
  1 96-95-79-62-static.hfc.comcastbusiness.net (96.95.79.62) 0 msec 4 msec 0 msec
  2 96.120.28.97 8 msec 8 msec 8 msec
  3 be92-sur02.stjoseph.mi.sbend.comcast.net (68.85.204.29) 8 msec 12 msec 12 msec
  4 68.85.183.237 12 msec 8 msec 8 msec
  5 be-126-ar01.area4.il.chicago.comcast.net (68.87.230.249) 12 msec 16 msec 16 msec
  6  *  *  *
  7  *  *  *
  8 LYNX-NETWOR.ear3.Chicago2.Level3.net (4.35.108.250) 20 msec 16 msec 16 msec
  9 66.35.87.172.in-addr.arpa.lynxnetworkgroup.com (172.87.35.66) 16 msec 28 msec 20 msec
10  *  *  *
11  *  *  *

Tracing the route to 172.87.34.1
VRF info: (vrf in name/id, vrf out name/id)
  1 173-15-238-74-static.hfc.comcastbusiness.net (173.15.238.74) 0 msec 4 msec 4 msec
  2 96.120.42.165 16 msec 16 msec 16 msec
  3 68.85.49.133 16 msec 12 msec 12 msec
  4 xe-0-0-0-0-sur01.stclairshors.mi.michigan.comcast.net (68.87.189.38) 16 msec 12 msec 12 msec
  5 te-0-3-0-15-ar02.pontiac.mi.michigan.comcast.net (68.85.235.249) 20 msec 20 msec 16 msec
  6 ae2.bar1.detroit1.level3.net (4.68.71.33) 44 msec 20 msec 16 msec
  7  *  *  *
  8 lynx-networ.ear3.chicago2.level3.net (4.35.108.250) 24 msec 32 msec 28 msec
  9 66.35.87.172.in-addr.arpa.lynxnetworkgroup.com (172.87.34.1) 32 msec *  32 msec

Now, this morning we're seeing an inability to pass two-way traffic over CenturyLink through Chicago:

  1   23ms    23ms    24ms    63.149.1.22
  2   *       *       *
  3   23ms    23ms    23ms    63.146.27.18
  4   *       *       *
  5   28ms    29ms    29ms    4.35.108.250
  6   35ms    28ms    29ms    172.87.34.1

Tracing the route to 63.149.1.22
VRF info: (vrf in name/id, vrf out name/id)
  1 172.87.35.66 0 msec 0 msec 0 msec
  2 6-2-40.ear3.chicago2.level3.net (4.35.108.249) 34 msec 0 msec 0 msec
  3 ae-14-3604.edge4.chicago3.level3.net (4.69.204.166) 8 msec 9 msec 8 msec
  4 chp-brdr-03.inet.qwest.net (63.146.27.17) 9 msec 8 msec 8 msec
 5 cer-edge-15.inet.qwest.net (205.171.93.26) 9 msec 8 msec 9 msec
  6 63.149.1.22 33 msec 34 msec *

Comcast appears blind to the issues we've been seeing at their edge.  Seems more like a Level 3 issue.  What's going on in Chicago lately?

--
B.J. Ash
bjash at kentrecords.com<mailto:bjash at kentrecords.com>
IT Supervisor | Kent Record Management Inc.
1950 Waldorf St. NW, Ste. A | Grand Rapids, MI  49544
W:616-459-6681 | www.kentrecords.com<http://www.kentrecords.com/>
Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Kent Record Management Inc. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Please contact the sender if you believe you have received this email in error.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20170511/6b067e94/attachment.htm>