[outages] Comcast/Level 3 Issues - Detroit/Chicago Areas -- Now CenturyLink today too?

Fri May 12 07:05:28 EDT 2017

I've been seeing this same behavior. Connection bouncing, sometimes multiple times, in the middle of the night. Had one branch office in IL bounce 3 times night before last.

Thanks!
--
Rob Hall, CCNP R&S, CCP-V
I.T. Manager
@Work Personnel Service
3815 Hwy 66 S, Suite 4
Rogersville, TN 37857
Office: (423)272-7897, x10113
Mobile: (423)312-6003
[cisco-certified-ccnp][CCP-V]

From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Dave Wagner via Outages
Sent: Thursday, May 11, 2017 2:22 PM
To: outages at outages.org
Subject: Re: [outages] Comcast/Level 3 Issues - Detroit/Chicago Areas -- Now CenturyLink today too?

We have been seeing more issues in the last few weeks with our Comcast Business customers also. Lots of offline alerts at night. Most in the same geographical area and it seems like they are doing different areas each week.

No evidence though from them. When we call them there like what are you talking about?

David Wagner | Director of Technology
Xpert Technologies, Inc.
  Office | 248.524.0101 x310
  Support | 248.524.0101 Opt 2
  Support | support at xxpert.com<mailto:support at xxpert.com>
  Web | xxpert.com <http://www.xxpert.com/>
[X_logo_tag (3)]

From: Outages [mailto:outages-bounces at outages.org] On Behalf Of B.J. Ash via Outages
Sent: Thursday, May 11, 2017 10:07 AM
To: Outages Mailing List <outages at outages.org<mailto:outages at outages.org>>
Subject: [outages] Comcast/Level 3 Issues - Detroit/Chicago Areas -- Now CenturyLink today too?

Does anyone have any information on some ongoing issues with Comcast in the Michigan area since end of April that involve intermittent packet loss, latency spikes (from normal 20 ms to 500+ ms), and just overall cruddy paths?

We're seeing issues with 3 of our sites in West/Central Michigan that use Comcast Business for Internet as they route from Comcast to Level 3.  Traceroutes seem to die somewhere in Chicago or leaving Detroit either in Level 3 or at the edge of Comcast:

Tracing the route to 172.87.34.1
VRF info: (vrf in name/id, vrf out name/id)
  1 50-77-204-206-static.hfc.comcastbusiness.net (50.77.204.206) 52 msec 40 msec 32 msec
  2 96.120.42.37 32 msec 44 msec 40 msec
  3 xe-7-0-6-sur02.muskegon.mi.michigan.comcast.net (68.85.189.25) 52 msec 52 msec 56 msec
  4 xe-1-3-1-0-sur01.lyndon.mi.michigan.comcast.net (68.87.189.177) 60 msec 68 msec 64 msec
  5 ae2.bar1.detroit1.level3.net (4.68.71.33) 72 msec 64 msec 64 msec
  6  *  *  *
  7 lynx-networ.ear3.chicago2.level3.net (4.35.108.250) 136 msec 144 msec 152 msec
  8 66.35.87.172.in-addr.arpa.lynxnetworkgroup.com (172.87.34.1) 148 msec *  220 msec

Tracing the route to 172.87.34.1
VRF info: (vrf in name/id, vrf out name/id)
  1 96-95-79-62-static.hfc.comcastbusiness.net (96.95.79.62) 0 msec 4 msec 0 msec
  2 96.120.28.97 8 msec 8 msec 8 msec
  3 be92-sur02.stjoseph.mi.sbend.comcast.net (68.85.204.29) 8 msec 12 msec 12 msec
  4 68.85.183.237 12 msec 8 msec 8 msec
  5 be-126-ar01.area4.il.chicago.comcast.net (68.87.230.249) 12 msec 16 msec 16 msec
  6  *  *  *
  7  *  *  *
  8 LYNX-NETWOR.ear3.Chicago2.Level3.net (4.35.108.250) 20 msec 16 msec 16 msec
  9 66.35.87.172.in-addr.arpa.lynxnetworkgroup.com (172.87.35.66) 16 msec 28 msec 20 msec
10  *  *  *
11  *  *  *

Tracing the route to 172.87.34.1
VRF info: (vrf in name/id, vrf out name/id)
  1 173-15-238-74-static.hfc.comcastbusiness.net (173.15.238.74) 0 msec 4 msec 4 msec
  2 96.120.42.165 16 msec 16 msec 16 msec
  3 68.85.49.133 16 msec 12 msec 12 msec
  4 xe-0-0-0-0-sur01.stclairshors.mi.michigan.comcast.net (68.87.189.38) 16 msec 12 msec 12 msec
  5 te-0-3-0-15-ar02.pontiac.mi.michigan.comcast.net (68.85.235.249) 20 msec 20 msec 16 msec
  6 ae2.bar1.detroit1.level3.net (4.68.71.33) 44 msec 20 msec 16 msec
  7  *  *  *
  8 lynx-networ.ear3.chicago2.level3.net (4.35.108.250) 24 msec 32 msec 28 msec
  9 66.35.87.172.in-addr.arpa.lynxnetworkgroup.com (172.87.34.1) 32 msec *  32 msec

Now, this morning we're seeing an inability to pass two-way traffic over CenturyLink through Chicago:

  1   23ms    23ms    24ms    63.149.1.22
  2   *       *       *
  3   23ms    23ms    23ms    63.146.27.18
  4   *       *       *
  5   28ms    29ms    29ms    4.35.108.250
  6   35ms    28ms    29ms    172.87.34.1

Tracing the route to 63.149.1.22
VRF info: (vrf in name/id, vrf out name/id)
  1 172.87.35.66 0 msec 0 msec 0 msec
  2 6-2-40.ear3.chicago2.level3.net (4.35.108.249) 34 msec 0 msec 0 msec
  3 ae-14-3604.edge4.chicago3.level3.net (4.69.204.166) 8 msec 9 msec 8 msec
  4 chp-brdr-03.inet.qwest.net (63.146.27.17) 9 msec 8 msec 8 msec
 5 cer-edge-15.inet.qwest.net (205.171.93.26) 9 msec 8 msec 9 msec
  6 63.149.1.22 33 msec 34 msec *

Comcast appears blind to the issues we've been seeing at their edge.  Seems more like a Level 3 issue.  What's going on in Chicago lately?

--
B.J. Ash
bjash at kentrecords.com<mailto:bjash at kentrecords.com>
IT Supervisor | Kent Record Management Inc.
1950 Waldorf St. NW, Ste. A | Grand Rapids, MI  49544
W:616-459-6681 | www.kentrecords.com<http://www.kentrecords.com/>
Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Kent Record Management Inc. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Please contact the sender if you believe you have received this email in error.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20170512/811fd6e1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 3689 bytes
Desc: image003.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20170512/811fd6e1/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 14979 bytes
Desc: image004.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20170512/811fd6e1/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2614 bytes
Desc: image001.jpg
URL: <https://puck.nether.net/pipermail/outages/attachments/20170512/811fd6e1/attachment.jpg>