[outages] Comcast Ashburn Backbone Issues

[Biddle, Josh]

Yes I have two customers over there – one in VA and the other WV.  I did not do traceroutes but I was observing strange activity.  Both the PE and the CPE would respond to ICMP but VPN tunnel to corporate office down and my monitoring was down.  I did notice the outages on downdetector over several states so I figured I would give it some time to resolve itself.  My two offices just returned about 10 minutes ago.  Thanks for the info about the routing loop.  The Comcast rep I spoke with seemed oblivious to the situation.  We’ll see what happens.

Best Regards,
Josh
From: Outages <outages-bounces at outages.org> On Behalf Of Geoff AB3LS via Outages
Sent: Wednesday, January 16, 2019 3:23 AM
To: outages at outages.org
Subject: [outages] Comcast Ashburn Backbone Issues

I am seeing what looks like a possible routing loop or some other routing issue going on with Comcast’s backbone in Ashburn, VA. I am a customer in Western Pennsylvania and I am unable to reach certain destinations such as www.duckduckgo.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.duckduckgo.com&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=60fytUEAoLJ26Q0HrYLYU_9b_JqnNKUvkwND0xLoGn8&s=kcN5pYRApDBJDuJ72gzNTQ0Ghx10PC2QCgA7q5PE2IA&e=>or www.google.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.google.org&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=60fytUEAoLJ26Q0HrYLYU_9b_JqnNKUvkwND0xLoGn8&s=Z9zYeX8Ekk-811XR7KLsKzb6CcBXyIJfRZxuTxCY5rQ&e=>. This has been going on for about an hour afaik. Traceroutes are playing ping pong between cr2.ashburn.va.ibone.comcast.netand either cs2 or cs4.ashburn.va.ibone.comcast.net

An example MTR to www.duckduckgo.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.duckduckgo.com&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=60fytUEAoLJ26Q0HrYLYU_9b_JqnNKUvkwND0xLoGn8&s=kcN5pYRApDBJDuJ72gzNTQ0Ghx10PC2QCgA7q5PE2IA&e=>.

HOST: gravitron Loss% Snt Last Avg Best Wrst StDev
1.|-- IRIDIUM 0.0% 100 0.7 0.9 0.6 6.8 0.7
2.|-- 96.120.62.193 0.0% 100 8.0 9.4 7.4 15.3 1.9
3.|-- 96.108.98.65 0.0% 100 8.9 9.7 7.3 19.4 2.2
4.|-- 162.151.65.5 0.0% 100 9.8 10.3 5.6 17.3 1.9
5.|-- be-7016-cr02.ashburn.va.i 0.0% 100 18.5 17.3 15.1 23.9 1.8
6.|-- be-1402-cs04.ashburn.va.i 98.0% 100 15.2 15.3 15.2 15.5 0.2
7.|-- be-1402-cr02.ashburn.va.i 97.0% 100 22.1 18.7 16.4 22.1 3.0
8.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
9.|-- be-1402-cr02.ashburn.va.i 94.0% 100 18.2 17.7 16.2 21.4 1.9
10.|-- be-1402-cs04.ashburn.va.i 99.0% 100 18.3 18.3 18.3 18.3 0.0
11.|-- be-1402-cr02.ashburn.va.i 94.0% 100 18.7 20.2 16.1 30.4 5.1
12.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
13.|-- be-1402-cr02.ashburn.va.i 93.9% 98 18.1 18.6 16.8 24.2 2.8
14.|-- ??? 100.0 96 0.0 0.0 0.0 0.0 0.0
15.|-- be-1402-cr02.ashburn.va.i 96.9% 96 16.7 17.6 16.7 18.7 1.0
16.|-- ??? 100.0 83 0.0 0.0 0.0 0.0 0.0
17.|-- be-1402-cr02.ashburn.va.i 95.2% 83 16.9 18.5 16.9 20.5 1.9
18.|-- ??? 100.0 65 0.0 0.0 0.0 0.0 0.0
19.|-- be-1402-cr02.ashburn.va.i 96.9% 65 17.5 17.0 16.6 17.5 0.6
20.|-- ??? 100.0 54 0.0 0.0 0.0 0.0 0.0

I have no way of knowing if this is isolated to the mid-Atlantic region or if it’s a widespread issue. Interestingly enough, Downdetector is showing a large spike in down reports across a wide range of services all at the same time.

Is anyone else experiencing this?

--
Geoff
AB3LS
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com immediately and permanently delete this email and any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20190116/329498b8/attachment.htm>