[Outages-discussion] Verizon slowdown between 1130 and 1330 eachday this week

[Wallace Keith]

I'm also coming out of boston after a couple of prior hops on
xl4.bos4.alter.net and we monitor latency to  a bunch of gomez sites
that monitor us.
The latency to those sites via uunet/alternet to those in Seattle and
DFW are flat all day long. (the rest route over a different carrier) You
should probably post a trace route taken during the slowdown,  showing
where you think the issue is.

  6     6 ms     7 ms     6 ms  0.so-0-0-2.xl3.bos4.alter.net
[152.63.16.250]
  7    14 ms    13 ms    13 ms  0.so-6-3-0.xl3.nyc4.alter.net
[152.63.0.69]
  8    17 ms    15 ms    17 ms  204.255.173.54
  9    15 ms    15 ms    15 ms  vlan52.ebr2.newyork2.level3.net
[4.69.138.254]
 10    45 ms    35 ms    36 ms  ae-2-2.ebr1.chicago1.level3.net
[4.69.132.65]
 11    33 ms    33 ms    33 ms  ae-13-55.car3.chicago1.level3.net
[4.68.101.135]
 12    33 ms    33 ms    33 ms  xe-0-3-0.cr2.ord1.us.nlayer.net
[4.71.101.14]
 13    39 ms    35 ms    35 ms  po6.ar2.ord1.us.scnet.net [69.31.111.6]
 14    36 ms    36 ms    36 ms  ge0-50.aggr4302.ord2.us.scnet.net
[75.102.0.114]
 15    35 ms    35 ms    35 ms  gimli.audiovideoweb.com [216.246.4.18]



-Keith


-----Original Message-----
From: outages-discussion-bounces at outages.org
[mailto:outages-discussion-bounces at outages.org] On Behalf Of Dale Amon
Sent: Wednesday, May 20, 2009 12:22 PM
To: outages-discussion at outages.org
Cc: Jim Caggy
Subject: [Outages-discussion] Verizon slowdown between 1130 and 1330
eachday this week

Has anyone else seen anything like this? Here's a report from our
network engineer on site:

> Hey Dale for the last couple days we've had either a major slowdown, 
> lost packets or completely lost TCP connections somewhere between the 
> hotel's endpoint and the Internet.  Verizon Tech support is being 
> stupid and just showing us the stats between the Hotels border router 
> to their gateway router, which is just a single hop, but there are 
> multiple hops within the Verizon backbone before our packets get to
the Internet.

> The problem starts between 1130 and 1330.  It's a major slowdown that
I
> would imagine other customers of Verizon are experiencing.   Is there
any
> resources you can tap to find out why we are getting this slowdown in 
> the backbone or on the way to the Internet.  Its definitely causing 
> major packet loss because its terminated our windows media streams 
> several times in the last few days during that time period.

> Here is a traceroute from us to the streaming provider so you can see 
> the hops.  There has to be others who have experienced problems along 
> this route during the timeframe mentioned above in the last few days.

  1    <1 ms    <1 ms    <1 ms  10.5.4.2 
  2     2 ms     1 ms     1 ms  65.206.112.1 
  3     3 ms     3 ms     3 ms  Serial5-1.GW1.BOS4.ALTER.NET
[157.130.11.161] 
  4     4 ms     4 ms     3 ms  167.at-6-0-0.XR2.BOS4.ALTER.NET
[152.63.16.242] 
  5     3 ms     3 ms     3 ms  0.so-3-0-2.XL4.BOS4.ALTER.NET
[152.63.16.133] 
  6    11 ms    11 ms    11 ms  0.so-6-3-0.XL4.NYC4.ALTER.NET
[152.63.0.73] 
  7    11 ms    11 ms    11 ms  204.255.173.54 
  8    16 ms    11 ms    11 ms  vlan52.ebr2.NewYork2.Level3.net
[4.69.138.254] 
  9    35 ms    36 ms    53 ms  ae-2-2.ebr1.Chicago1.Level3.net
[4.69.132.65] 
 10    33 ms    33 ms    34 ms  ae-13-51.car3.Chicago1.Level3.net
[4.68.101.7] 
 11    97 ms    33 ms    33 ms  xe-0-3-0.cr2.ord1.us.nlayer.net
[4.71.101.14] 
 12    33 ms    35 ms    33 ms  po6.ar2.ord1.us.scnet.net [69.31.111.6] 
 13    35 ms    34 ms    35 ms  ge0-50.aggr4302.ord2.us.scnet.net
[75.102.0.114] 
 14    33 ms    32 ms    33 ms  gimli.audiovideoweb.com [216.246.4.18]

Thanks for any help. It's got us at the
hair pulling stage. Given what we're using it for your own C-level
management might be there ;-) (it's the JPMorgan Tech09 Investor
conference).