[outages] Cogent routing issues, possibly SoCal?

Mon Sep 19 15:42:27 EDT 2016

We have had Cogent outages for 3 straight days.  Links are currently flapping again in Las Vegas.  Here is the RFO from 2 days ago…

“Dear Cogent Customer,

This letter is in response to the Cogent Communications network events that occurred on September 16, 2016 affecting Phoenix area customers.

  On September 16, 2016 one of our local fiber vendors for our Phoenix market performed a scheduled maintenance to relocate fibers at our main Hub site. Unfortunately, the scope of the maintenance forwarded to us by our provider was missing information in relation to our backbone fibers. We were not made aware that our redundant links would be impacted concurrently. The maintenance isolated parts of our Phoenix market between 3:16 AM ET and 7:11 AM ET. Additionally our vendor had issues with the fibers for one of our metro rings that kept some customers down until approximately 4:00 PM ET.”

From: Outages <outages-bounces at outages.org> on behalf of SecurityPatches via Outages <outages at outages.org>
Reply-To: SecurityPatches <securitypatches at indataipm.com>
Date: Monday, September 19, 2016 at 12:22 PM
To: "'outages at outages.org'" <outages at outages.org>
Subject: Re: [outages] Cogent routing issues, possibly SoCal?

From our status board:
[cid:image001.png at 01D21273.47147DF0]
[cid:image002.png at 01D21273.47147DF0]
Richard Walz | INDATA
Director of IT
rwalz at indataipm.com<mailto:rwalz at indataipm.com> | (203) 302-7114<tel:12033027114>

From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Mason Moody via Outages
Sent: Monday, September 19, 2016 3:08 PM
To: outages at outages.org
Subject: Re: [outages] Cogent routing issues, possibly SoCal?

I wonder if the issues around Los Angeles are having knock-on effects further out. I am seeing traffic from San Francisco going spotty:

HOST: bespin                                            Loss%   Snt   Last   Avg  Best  Wrst StDev

  3.|-- te0-0-1-3.agr12.sjc01.atlas.cogentco.com           0.0%    50    1.2   1.3   1.1   3.6   0.4
  4.|-- te0-7-0-21.ccr21.sjc01.atlas.cogentco.com          0.0%    50    1.0   1.2   0.9   2.6   0.3
  5.|-- be2164.ccr21.sfo01.atlas.cogentco.com             58.0%    50  372.3 374.1 363.2 389.1   7.0
  6.|-- be2086.ccr21.slc01.atlas.cogentco.com             32.0%    50  424.3 430.0 413.2 473.4  16.3
  7.|-- be3038.ccr22.den01.atlas.cogentco.com             94.0%    50   28.2  28.3  28.2  28.4   0.0
  8.|-- be3036.ccr22.mci01.atlas.cogentco.com             88.0%    50   40.1  40.0  39.7  40.1   0.0
  9.|-- be2832.ccr42.ord01.atlas.cogentco.com             92.0%    50   51.9  52.0  51.9  52.1   0.0
 10.|-- be2736.ccr42.atl01.atlas.cogentco.com             98.0%    50   73.2  73.2  73.2  73.2   0.0
 11.|-- be2785.rcr12.jax01.atlas.cogentco.com             18.0%    50   77.5  79.7  77.5  83.0   1.2
 12.|-- te0-0-2-0.rcr12.mco01.atlas.cogentco.com          26.0%    50   81.0  83.0  80.8  84.3   1.0
 13.|-- te0-0-1-3.nr11.b006655-1.mco01.atlas.cogentco.com 24.0%    50   81.9  83.8  81.7  85.5   1.1
 14.|-- 38.104.89.122                                     16.0%    50   80.5  83.4  80.4  98.6   2.9
 15.|-- 209.208.7.42                                      24.0%    50   80.6  95.4  80.6 545.4  75.0
 16.|-- 209.208.27.233                                   30.0%    50   81.4  84.5  81.4  86.0   1.3

On 2016-09-19 14:55, Joe Klein via Outages wrote:

http://status.cogentco.com/

Cogent Network Status Report Last Updated Mon Sep 19 18:52:09 2016 GMT

The Cogent Network Operations Center is currently investigating an outage in the Los Angeles area. The master case is HD6655601. There is no estimated time to repair.

Some customers in the Raleigh North Carolina area may be experiencing a loss of connectivity. Engineers are currently investigating but there is no estimated time to repair at this time. The master case is HD6655487

--
____________

Mason Moody
Network Security Engineer
Atlantic.Net
Phone: 800-422-2936 x4431
Int'l: +1-321-206-3731

[cid:image003.gif at 01D21273.47147DF0]
This e-mail, including any attachments, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of any part of this e-mail is strictly prohibited; please contact the sender and permanently delete the original and any copies of it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20160919/4eb871d8/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 33627 bytes
Desc: image001.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20160919/4eb871d8/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 35994 bytes
Desc: image002.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20160919/4eb871d8/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 1114 bytes
Desc: image003.gif
URL: <https://puck.nether.net/pipermail/outages/attachments/20160919/4eb871d8/attachment.gif>