[outages] Road Runner/Level 3 broked?

[Darrell Hyde]

Were they able to provide any detail with respect to root cause?
________________________________________
From: Outages [outages-bounces at outages.org] On Behalf Of Darren Schreiber [darren at 2600hz.com]
Sent: Saturday, October 19, 2013 12:01 PM
To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List
Subject: Re: [outages] Road Runner/Level 3 broked?

Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates.

Appears the ETA is within the next 2 hours.

From: Darren Schreiber <darren at 2600hz.com<mailto:darren at 2600hz.com>>
Date: Saturday, October 19, 2013 7:03 AM
To: Tim Glen <tglen at asicentral.com<mailto:tglen at asicentral.com>>, 'Sajal Kayan' <sajal83 at gmail.com<mailto:sajal83 at gmail.com>>, OUTAGES Mailing List <Outages at outages.org<mailto:Outages at outages.org>>
Subject: Re: [outages] Road Runner/Level 3 broked?

I can confirm the same from our datacenter in NY. Tracepath goes in a loop…

From: Tim Glen <tglen at asicentral.com<mailto:tglen at asicentral.com>>
Date: Saturday, October 19, 2013 6:59 AM
To: 'Sajal Kayan' <sajal83 at gmail.com<mailto:sajal83 at gmail.com>>, OUTAGES Mailing List <Outages at outages.org<mailto:Outages at outages.org>>
Subject: Re: [outages] Road Runner/Level 3 broked?

I just opened a ticket with L3.

I've been having packet loss since 9:07AM today over my L3 circuit.

I am in Philly and L3's customer access router is experiencing high packet loss.



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Timothy Glen
Sr. Network Engineer
Advertising Specialty Institute | Promoting Success

tglen at asicentral.com<mailto:tglen at asicentral.com>
tel: (215) 953-3692
fax: (215) 953-3759
Think before you print.



________________________________
From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Sajal Kayan
Sent: Saturday, October 19, 2013 9:52 AM
To: OUTAGES Mailing List
Subject: [outages] Road Runner/Level 3 broked?

Using road runner in Brooklyn NY.

Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3.

(FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> )

sajal at sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com>
HOST: sajal-laptop                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10    1.2 110.3   1.0 603.9 230.5
  2.|-- cpe-24-90-104-1.nyc.res.r  0.0%    10   37.9 207.9  17.0 1418. 444.2
  3.|-- tge-0-0-0-1.nycmnyr-rtr2.  0.0%    10    9.8  93.3   9.4 471.9 170.3
  4.|-- agg23.nyquny9101r.nyc.rr.  0.0%    10   12.6  78.3  10.1 402.2 142.2
  5.|-- 107.14.19.22               0.0%    10   13.8  64.6  11.5 331.0 112.6
  6.|-- ae-0-0.pr0.nyc30.tbone.rr  0.0%    10   13.8  50.9  11.4 257.6  82.3
  7.|-- xe-4-2-0.edge4.frankfurt1  0.0%    10   13.4  35.0   9.1 186.5  55.8
  8.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
sajal at sajal-laptop:~$


sajal at sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com>
HOST: sajal-laptop                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10    1.2   2.9   1.0   9.5   2.8
  2.|-- cpe-24-90-104-1.nyc.res.r  0.0%    10  151.6 184.3  75.8 529.8 137.6
  3.|-- tge-0-0-0-1.nycmnyr-rtr2.  0.0%    10  252.6 160.8  64.7 299.7  87.2
  4.|-- agg23.nyquny9101r.nyc.rr.  0.0%    10  243.5 166.4  64.1 345.8  88.3
  5.|-- ae-3-0.cr0.nyc20.tbone.rr  0.0%    10  246.7 189.2  88.6 309.1  75.7
  6.|-- 107.14.17.169              0.0%    10  256.6 175.3 102.6 256.6  60.5
  7.|-- xe-4-2-0.edge4.frankfurt1  0.0%    10  258.6 179.6  95.4 302.2  69.1
  8.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
sajal at sajal-laptop:~$

sajal at sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com>
HOST: sajal-laptop                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10    4.0   3.1   1.1   8.5   2.8
  2.|-- cpe-24-90-104-1.nyc.res.r  0.0%    10  225.9 163.7  97.8 252.8  57.0
  3.|-- tge-0-0-0-1.nycmnyr-rtr2.  0.0%    10  298.0 168.6  69.3 298.0  79.0
  4.|-- agg23.nyquny9101r.nyc.rr.  0.0%    10  292.5 155.0  90.8 292.5  71.0
  5.|-- ae-3-0.cr0.nyc20.tbone.rr  0.0%    10  239.0 148.6  68.5 248.6  63.2
  6.|-- 107.14.17.169              0.0%    10  268.1 148.7  73.0 268.1  64.9
  7.|-- xe-4-2-0.edge4.frankfurt1 10.0%    10  235.1 130.2  66.2 235.1  49.2
  8.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
sajal at sajal-laptop:~$

-Sajal