[outages] Chicago-area Level3 to Amazon AWS EAST-1

[Sarlas, George]

Since approx. 4am CT, we've been seeing 15-30% dropped packets for traffic going from our Chicago Level3 connection to Amazon's EAST-1 data center.  Our San Francisco office (also using Level3) isn't experiencing a problem.  I've re-routed this traffic to our backup Internet connection (Airlogic) for now, and that traffic is getting through just fine.  Anyone else seeing this problem?  Ticket already opened with Level3, waiting to hear back.  Thank you.

Here's the route my traffic is taking (time outs at the bottom are normal, ICMP gets blocked from that point on).





Tracing route to zioreports.com [75.101.163.221]

over a maximum of 30 hops:



  1     1 ms     1 ms     1 ms  ge-6-2-107.car4.Chicago1.Level3.net [4.71.102.161]

  2     *        *        *     Request timed out.

  3    16 ms    22 ms    25 ms  ae-5-5.ebr2.Chicago2.Level3.net [4.69.140.194]

  4    19 ms    24 ms    25 ms  ae-6-6.ebr2.Washington12.Level3.net [4.69.148.145]

  5     *        *        *     Request timed out.

  6    16 ms    18 ms    23 ms  ae-82-82.csw3.Washington1.Level3.net [4.69.134.154]

  7    16 ms    16 ms    17 ms  ae-3-80.edge2.Washington1.Level3.net [4.69.149.142]

  8    17 ms    17 ms    16 ms  AMAZON.COM.edge2.Washington1.Level3.net [4.79.22.74]

 9     *        *        *     Request timed out.

10    17 ms    18 ms    17 ms  72.21.222.139

11    18 ms    18 ms    18 ms  216.182.224.17

12     *        *        *     Request timed out.

13     *        *        *     Request timed out.

14     *        *        *     Request timed out.

15     *        *        *     Request timed out.

16     *        *        *     Request timed out.

17     *        *        *     Request timed out.

18     *        *     ^C


Here are the results of a TCPing to port 443:


2012:09:27 09:50:02 Probing 75.101.163.221:443/tcp - Port is open - time=20.926ms

2012:09:27 09:50:03 Probing 75.101.163.221:443/tcp - Port is open - time=31.168ms

2012:09:27 09:50:05 Probing 75.101.163.221:443/tcp - Socket is not connected (10057) - time=2012.341ms

2012:09:27 09:50:07 Probing 75.101.163.221:443/tcp - Port is open - time=31.358ms

2012:09:27 09:50:08 Probing 75.101.163.221:443/tcp - Socket is not connected (10057) - time=2012.448ms

2012:09:27 09:50:10 Probing 75.101.163.221:443/tcp - Port is open - time=30.985ms

2012:09:27 09:50:12 Probing 75.101.163.221:443/tcp - Port is open - time=31.224ms

2012:09:27 09:50:13 Probing 75.101.163.221:443/tcp - Port is open - time=32.065ms

2012:09:27 09:50:15 Probing 75.101.163.221:443/tcp - Socket is not connected (10057) - time=2012.375ms

2012:09:27 09:50:17 Probing 75.101.163.221:443/tcp - Port is open - time=31.115ms

Ping statistics for 75.101.163.221:443

     10 probes sent.

     7 successful, 3 failed.

Approximate trip times in milli-seconds (successful connections only):
    Minimum = 20.926ms, Maximum = 32.065ms, Average = 20.884ms



----
George Sarlas
Manager, IT Operations
iRhythm Technologies, Inc.

2 Marriott Dr.
Lincolnshire, IL 60069

email: gsarlas at irhythmtech.com
phone: 224-543-4253

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20120927/7ba36f99/attachment.htm>