[outages] Road Runner/Level 3 broked?

[Mitch]

Most recent update we have which includes the "external ticket" which
i think is the master

*** CASCADED EXTERNAL NOTES 20-Oct-2013 03:42:42 GMT From CASE: 7126134 - Event
The Transport NOC has advised that the replacement circuit pack has
arrived on site, and in cooperation the field technician will replace
the faulty circuit pack. Once the stability of the circuit pack is
confirmed, efforts will begin to move all services back to their
intended paths. An update will be sent prior to the commencement of
that demand maintenance activity.

On Sat, Oct 19, 2013 at 7:53 PM, Adam Greene <maillist at webjogger.net> wrote:
> Thanks Tim and Darrell, these updates are priceless for us, since the
> provider we use to connect to Level3 still seems to be totally clueless
> about the issue.
>
>
>
> Would you mind sharing the Level3 master ticket # so we can raise their
> awareness?
>
>
>
> If you receive any further updates, especially confirmation of issue
> resolution, it would be wonderful if you post that to this list, too.
>
>
>
> Thanks again.
>
>
>
>
>
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Tim Glen
> Sent: Saturday, October 19, 2013 7:20 PM
> To: Darrell Hyde
> Cc: OUTAGES Mailing List
>
>
> Subject: Re: [outages] Road Runner/Level 3 broked?
>
>
>
> Hardware failure and apparently the part that was sent to fix was also
> broke...  This is from the most recent update to the ticket.
>
>
> The Transport NOC has reported that the field technician completed rerouting
> traffic; however, some services failed to restore. The replacement circuit
> pack is still in transit to the site, and an emergency maintenance window
> has been scheduled for 04:00 GMT to perform the replacement. Once the
> circuit pack is replaced, customers will be reverted to the original route.
> All services will be impacted while traffic is being routed back. The next
> update will be sent once it has been confirmed that the maintenance activity
> has commenced, or should any additional information become available.
>
>
> On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <DHyde at hosting.com> wrote:
>
> Were they able to provide any detail with respect to root cause?
> ________________________________________
> From: Outages [outages-bounces at outages.org] On Behalf Of Darren Schreiber
> [darren at 2600hz.com]
> Sent: Saturday, October 19, 2013 12:01 PM
> To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List
> Subject: Re: [outages] Road Runner/Level 3 broked?
>
> Level3 just called me back for the ticket I filed. Tech was very
> knowledgable. Is aware of the issue (obviously) and has attached my ticket
> to the master event ticket so that I can see realtime updates.
>
> Appears the ETA is within the next 2 hours.
>
> From: Darren Schreiber <darren at 2600hz.com<mailto:darren at 2600hz.com>>
> Date: Saturday, October 19, 2013 7:03 AM
> To: Tim Glen <tglen at asicentral.com<mailto:tglen at asicentral.com>>, 'Sajal
> Kayan' <sajal83 at gmail.com<mailto:sajal83 at gmail.com>>, OUTAGES Mailing List
> <Outages at outages.org<mailto:Outages at outages.org>>
> Subject: Re: [outages] Road Runner/Level 3 broked?
>
> I can confirm the same from our datacenter in NY. Tracepath goes in a loop…
>
> From: Tim Glen <tglen at asicentral.com<mailto:tglen at asicentral.com>>
> Date: Saturday, October 19, 2013 6:59 AM
> To: 'Sajal Kayan' <sajal83 at gmail.com<mailto:sajal83 at gmail.com>>, OUTAGES
> Mailing List <Outages at outages.org<mailto:Outages at outages.org>>
> Subject: Re: [outages] Road Runner/Level 3 broked?
>
> I just opened a ticket with L3.
>
> I've been having packet loss since 9:07AM today over my L3 circuit.
>
> I am in Philly and L3's customer access router is experiencing high packet
> loss.
>
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Timothy Glen
> Sr. Network Engineer
> Advertising Specialty Institute | Promoting Success
>
> tglen at asicentral.com<mailto:tglen at asicentral.com>
> tel: (215) 953-3692
> fax: (215) 953-3759
> Think before you print.
>
>
>
> ________________________________
> From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Sajal Kayan
> Sent: Saturday, October 19, 2013 9:52 AM
> To: OUTAGES Mailing List
> Subject: [outages] Road Runner/Level 3 broked?
>
> Using road runner in Brooklyn NY.
>
> Cant access certain sites ( Facebook, twitter, citibank) . All of them have
> one thing in common, they go thru level3.
>
> (FYI full name of hop 7 is
> xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net>
> )
>
> sajal at sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com>
> HOST: sajal-laptop                Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1.|-- 192.168.1.1                0.0%    10    1.2 110.3   1.0 603.9 230.5
>  2.|-- cpe-24-90-104-1.nyc.res.r  0.0%    10   37.9 207.9  17.0 1418. 444.2
>  3.|-- tge-0-0-0-1.nycmnyr-rtr2.  0.0%    10    9.8  93.3   9.4 471.9 170.3
>  4.|-- agg23.nyquny9101r.nyc.rr.  0.0%    10   12.6  78.3  10.1 402.2 142.2
>  5.|-- 107.14.19.22               0.0%    10   13.8  64.6  11.5 331.0 112.6
>  6.|-- ae-0-0.pr0.nyc30.tbone.rr  0.0%    10   13.8  50.9  11.4 257.6  82.3
>  7.|-- xe-4-2-0.edge4.frankfurt1  0.0%    10   13.4  35.0   9.1 186.5  55.8
>  8.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
> sajal at sajal-laptop:~$
>
>
> sajal at sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com>
> HOST: sajal-laptop                Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1.|-- 192.168.1.1                0.0%    10    1.2   2.9   1.0   9.5   2.8
>  2.|-- cpe-24-90-104-1.nyc.res.r  0.0%    10  151.6 184.3  75.8 529.8 137.6
>  3.|-- tge-0-0-0-1.nycmnyr-rtr2.  0.0%    10  252.6 160.8  64.7 299.7  87.2
>  4.|-- agg23.nyquny9101r.nyc.rr.  0.0%    10  243.5 166.4  64.1 345.8  88.3
>  5.|-- ae-3-0.cr0.nyc20.tbone.rr  0.0%    10  246.7 189.2  88.6 309.1  75.7
>  6.|-- 107.14.17.169              0.0%    10  256.6 175.3 102.6 256.6  60.5
>  7.|-- xe-4-2-0.edge4.frankfurt1  0.0%    10  258.6 179.6  95.4 302.2  69.1
>  8.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
> sajal at sajal-laptop:~$
>
> sajal at sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com>
> HOST: sajal-laptop                Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1.|-- 192.168.1.1                0.0%    10    4.0   3.1   1.1   8.5   2.8
>  2.|-- cpe-24-90-104-1.nyc.res.r  0.0%    10  225.9 163.7  97.8 252.8  57.0
>  3.|-- tge-0-0-0-1.nycmnyr-rtr2.  0.0%    10  298.0 168.6  69.3 298.0  79.0
>  4.|-- agg23.nyquny9101r.nyc.rr.  0.0%    10  292.5 155.0  90.8 292.5  71.0
>  5.|-- ae-3-0.cr0.nyc20.tbone.rr  0.0%    10  239.0 148.6  68.5 248.6  63.2
>  6.|-- 107.14.17.169              0.0%    10  268.1 148.7  73.0 268.1  64.9
>  7.|-- xe-4-2-0.edge4.frankfurt1 10.0%    10  235.1 130.2  66.2 235.1  49.2
>  8.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
> sajal at sajal-laptop:~$
>
> -Sajal
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>