<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div style="RIGHT: auto"><SPAN style="RIGHT: auto">Sorry I was interested in outages not Jays mistakes. I really just need a site that reports outages as it effects business and not the details on how Jay fixed his issues.</SPAN></div>
<div style="RIGHT: auto"><SPAN style="RIGHT: auto"><VAR id=yui-ie-cursor></VAR> </SPAN></div>
<div><BR></div>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"><FONT size=2 face=Arial>
<DIV style="BORDER-BOTTOM: #ccc 1px solid; BORDER-LEFT: #ccc 1px solid; PADDING-BOTTOM: 0px; LINE-HEIGHT: 0; MARGIN: 5px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; HEIGHT: 0px; FONT-SIZE: 0px; BORDER-TOP: #ccc 1px solid; BORDER-RIGHT: #ccc 1px solid; PADDING-TOP: 0px" class=hr contentEditable=false readonly="true"></DIV><B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Jay Ashworth <jra@baylink.com><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> outages@outages.org<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Friday, September 23, 2011 7:17 PM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> [outages] Lessons Learned: RRTB outage<BR></FONT><BR>So I had to renumber some servers this afternoon, cause I was expanding to <BR>a larger netblock (a 28 instead of a 29).<BR><BR>I renumbered my servers and my DNS (which I'd set the TTL on to 300 like a<BR>good boy on Wednesday), and then pulled the trigger with Road Runner.
He<BR>"rescripted" his SMC router (the likely cause of some standard deviation noted<BR>by a couple of reporters -- the router, not the rescripting), and I pinged<BR>it and it was ok, and I mtr'd it and it was ok, so I hit the webserver,<BR>and that came up fine, too.<BR><BR>So then my boss calls me 15 minutes later: it's not working.<BR><BR>"I wonder what that could be", sez I; I'd even traced and hit the webserver<BR>from my Android phone (Sprint; Opera Mobile 11), and it had worked fine.<BR><BR>That was Red Herring #1.<BR><BR>So my boss uses a Mac. So does my best friend, and while he was on the way<BR>out the door to a second-anniversary-wake for a guy we went to school with,<BR>he took a moment to try to hit it as well. No luck.<BR><BR>That was Red Herring #2 (both of them use Macs).<BR><BR>Those of you who've been playing close, careful attention here may have<BR>noticed by now the thing I did *not* say I'd done: <BR><BR>Changing the
default gateway on the server.<BR><BR>My office lan could hit it *because its uplink was in the same network*;<BR>*it* had a route for that network. Everyone else... couldn't.<BR><BR>Apparently, Sprint operates a caching server, even if you're using the <BR>version of Opera (Mobile, not Mini) that does *not*, which explains Red<BR>Herring #1.<BR><BR>As for Red Herring #2, well... Macs don't, apparently, hard-cache IPs the<BR>way WinXP does (I'm looking at *you*, "ipconfig/ flushdns"), but I already<BR>knew that, because boss had the right address.<BR><BR>Lesson Learned: Make sure you know what your diagnostic tests are telling <BR>you, before you use them to rule out possible problems. Better yet: don't<BR>rule those potential problems out at all: work your whole diagnostic tree<BR>every time<BR><BR>Oh: I forgot Red Herring #3: the traces that broke *didn't hit that carrier<BR>edge router* for some reason. No clue why.<BR><BR>Thanks to
the dozen or so people who responded; a couple of whom have<BR>way too {much time,many servers} on their hands. :-)<BR><BR>Followups to -discuss<BR><BR>Cheers,<BR>-- jra<BR>-- <BR>Jay R. Ashworth Baylink <A href="mailto:jra@baylink.com" ymailto="mailto:jra@baylink.com">jra@baylink.com</A><BR>Designer The Things I Think RFC 2100<BR>Ashworth & Associates <A href="http://baylink.pitas.com/" target=_blank>http://baylink.pitas.com</A> 2000 Land Rover DII<BR>St Petersburg FL USA <A href="http://photo.imageinc.us/" target=_blank>http://photo.imageinc.us</A> +1 727 647
1274<BR>_______________________________________________<BR>Outages mailing list<BR><A href="mailto:Outages@outages.org" ymailto="mailto:Outages@outages.org">Outages@outages.org</A><BR><A href="https://puck.nether.net/mailman/listinfo/outages" target=_blank>https://puck.nether.net/mailman/listinfo/outages</A><BR><BR><BR></DIV></DIV></div></body></html>