[outages] Possible outage affecting Amazon.com?

Chris Hudson chris at htswireless.com
Fri Feb 21 20:01:05 EST 2014 

The main problem we were seeing is that the main page would only partially
load, and you could not browse to other pages. Although it does seem to be
working much better now.

Chris


-----Original Message-----
From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Jeremy
Chadwick
Sent: Friday, February 21, 2014 6:47 PM
To: Chris Stone
Cc: outages at outages.org
Subject: Re: [outages] Possible outage affecting Amazon.com?

What you're seeing is commonplace too.

I would say both ICMP TTL exceeded (used for route/path discovery) and
ECHO/ECHO REPLY look to be filtered at border routers, probably border
routers that Amazon owns/maintains.  While mtr by default uses ICMP
exclusively, ICMP TTL exceeded is still used for path discovery, hence
people thinking "there must be something wrong".

mtr does have a -T flag that uses TCP SYN instead of ICMP ECHO for endpoint
reachability, i.e. mtr -T -P 80 www.amazon.com, but all it does for me on
FreeBSD is bail out with a hard-to-read error ("bind(): No
error: 0") and doesn't reset tty params on failure, resulting in a wonderful
mess of a terminal (requires "stty icanon isig iexten echo icrnl ixon brkint
onlcr" be run before things begin behaving again).

But traceroute -P tcp -p 80 www.amazon.com doesn't improve things either,
going back to my theory that TTL exceeded is probably filtered.

Testing end-to-end like you did (in a browser, telnet port 80, etc.) is one
of the only things one can do in this situation.

<cry>
I love how network admins still to this day filter all this, making
troubleshooting virtually impossible.  The mentality I find hilarious; "if
we permit it, someone could ping -f the router's IP that responds to TTL
exceeded and cause high CPU" -- and if that doesn't work, guess what the
packet kids do?  Send 200gbit/sec of traffic at your uplink, effectively
accomplishing the same result of knocking you offline.

And no I will not respond to commentaries about this section of my mail.
</cry>

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Fri, Feb 21, 2014 at 05:13:45PM -0700, Chris Stone wrote:
> Am in Colorado on CenturyLink DSL. When I try and trace to 
> www.amazon.com, it fails:
> 
> Start: Fri Feb 21 17:07:37 2014
> HOST: orion                       Loss%   Snt   Last   Avg  Best Wrst
StDev
>   1.|--                            0.0%    10    0.4   0.4   0.3 0.5   0.0
>   2.|-- hlrn-dsl-gw07.hlrn.qwest.  0.0%    10   35.4  38.8  34.9 70.7
11.2
>   3.|-- hlrn-agw1.inet.qwest.net   0.0%    10   35.2  35.6  34.7 40.6
1.7
>   4.|-- dca2-edge-01.inet.qwest.n 50.0%    10   85.0  85.2  84.9 85.6
0.0
>   5.|-- ???                       100.0    10    0.0   0.0   0.0 0.0   0.0
> 
> 
> But the web site comes up fine in my browser. Checked IPs and tried 
> with that and still the same - site comes up fine but trace and pings 
> fail at
> CTL:
> 
> 17:08:49-cls at orion:~/scripts/ctr$ host www.amazon.com www.amazon.com 
> has address 72.21.215.232 17:09:40-cls at orion:~/scripts/ctr$ mtr 
> 72.21.215.232 17:09:49-cls at orion:~/scripts/ctr$ mtr --report 
> --report-cycles=10 72.21.215.232
> Start: Fri Feb 21 17:10:02 2014
> HOST: orion                     Loss%   Snt   Last   Avg  Best Wrst StDev
>   1.|--                            0.0%    10    0.5   0.5   0.4 1.3   0.0
>   2.|-- hlrn-dsl-gw07.hlrn.qwest.  0.0%    10   35.4  41.0  35.1 89.4
17.0
>   3.|-- hlrn-agw1.inet.qwest.net   0.0%    10   35.1  35.0  34.9 35.4
0.0
>   4.|-- dca2-edge-01.inet.qwest.n  0.0%    10   85.1  85.2  84.9 85.5
0.0
>   5.|-- ???                       100.0    10    0.0   0.0   0.0 0.0   0.0
> 17:10:17-cls at orion:~/scripts/ctr$ ping 72.21.215.232 PING 
> 72.21.215.232 (72.21.215.232) 56(84) bytes of data.
> ^C
> --- 72.21.215.232 ping statistics ---
> 1 packets transmitted, 0 received, 100% packet loss, time 0ms
> 
> Same going out our data center in Denver on Cogent:
> 
> HOST: pegasus                     Loss%   Snt   Last   Avg  Best Wrst
StDev
>   1. core                          0.0%     3    0.6   0.8   0.6 1.1   0.3
>   2. fa0-2.na01.b009854-0.den01.a  0.0%     3    0.9   1.8   0.9 2.9   1.0
>   3. vl3509.mag02.den01.atlas.cog  0.0%     3    1.9   1.3   0.9 1.9   0.5
>   4. te0-7-0-5.ccr21.den01.atlas.  0.0%     3    1.1   1.1   1.1 1.1   0.0
>   5. be2129.mpd21.mci01.atlas.cog  0.0%     3   13.1  13.0  12.8 13.1
0.1
>   6. be2064.ccr21.dfw01.atlas.cog  0.0%     3   22.8  23.0  22.8 23.3
0.3
>   7. be2031.ccr21.dfw03.atlas.cog  0.0%     3   23.5  23.9  23.5 24.2
0.4
>   8. ntt.dfw03.atlas.cogentco.com  0.0%     3   23.7  23.9  23.7 24.5
0.4
>   9. ae-1.r08.dllstx09.us.bb.gin.  0.0%     3   25.2  24.2  23.7 25.2
0.8
>  10. ???                          100.0     3    0.0   0.0   0.0 0.0   0.0
> 
> 
> 
> Chris
> 
> 
> On 02/21/2014 05:00 PM, Dj Padzensky wrote:
> >
> >Looks like (more) Level3 problems...
> >
> >--Dj
> >
> >On Feb 21, 2014, at 3:57 PM, Grant Ridder <shortdudey123 at gmail.com 
> ><mailto:shortdudey123 at gmail.com>> wrote:
> >
> >>No problems from comcast in Mountain View ca
> >>
> >>
> >>On Fri, Feb 21, 2014 at 3:47 PM, Clayton Dukes <cdukes at gmail.com 
> >><mailto:cdukes at gmail.com>> wrote:
> >>
> >>    I happen to be in Marrakech at the moment and am able to access it.
> >>
> >>    ______________________________________________________________
> >>
> >>    Clayton Dukes
> >>    ______________________________________________________________
> >>
> >>
> >>    On Sat, Feb 22, 2014 at 12:36 AM, <chris at htswireless.com
> >>    <mailto:chris at htswireless.com>> wrote:
> >>
> >>        Is anyone else seeing traffic routing all over the place trying
> >>        to get to
> >>        Amazon.com <http://Amazon.com>?
> >>
> >>
> >>        Chris
> >>
> >>
> >>        _______________________________________________
> >>        Outages mailing list
> >>        Outages at outages.org <mailto:Outages at outages.org>
> >>        https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >>
> >>    _______________________________________________
> >>    Outages mailing list
> >>    Outages at outages.org <mailto:Outages at outages.org>
> >>    https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >>_______________________________________________
> >>Outages mailing list
> >>Outages at outages.org <mailto:Outages at outages.org> 
> >>https://puck.nether.net/mailman/listinfo/outages
> >
> >
> >
> >_______________________________________________
> >Outages mailing list
> >Outages at outages.org
> >https://puck.nether.net/mailman/listinfo/outages
> 

> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

More information about the Outages mailing list