From virendra.rode at gmail.com Sat Mar 13 11:14:37 2010 From: virendra.rode at gmail.com (virendra rode) Date: Sat, 13 Mar 2010 08:14:37 -0800 Subject: [Outages-discussion] Interesting Read - Post-mortem for GoogleApp Engine on February 24th, 2010 outage Message-ID: <4B9BB9ED.2030702@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Headless chicken syndrome that took place during this outage, kinda what we all experience in our own environment. Yes there is lesson to be learned from this. Update your process and keep your management network robust so your health status dashboard is reachable to enable the whole story in a timely manner. There is a take away for all of us, take it for what its worth. The weekend is here, time to recharge, empty your head. Like Karl says, remember to clean up after yourselves and turn off the light when you're done. I almost forgot, this link is posted off http://wiki.outages.org -> Dashboard -> Lesson Learned Enjoy:) regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFLm7ntpbZvCIJx1bcRAnBZAJ9bbfjbUwsvxvNlZemYG8unVSjPNwCfSxu/ XMKz4by6UwE67fVWv4Ecp3k= =hCAC -----END PGP SIGNATURE----- From LarrySheldon at cox.net Sat Mar 13 12:44:34 2010 From: LarrySheldon at cox.net (Larry Sheldon) Date: Sat, 13 Mar 2010 11:44:34 -0600 Subject: [Outages-discussion] Interesting Read - Post-mortem for GoogleApp Engine on February 24th, 2010 outage In-Reply-To: <4B9BB9ED.2030702@gmail.com> References: <4B9BB9ED.2030702@gmail.com> Message-ID: <4B9BCF02.8090600@cox.net> On 3/13/2010 10:14, virendra rode wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, > > Headless chicken syndrome that took place during this outage, kinda what > we all experience in our own environment. > > Yes there is lesson to be learned from this. Update your process and > keep your management network robust so your health status dashboard is > reachable to enable the whole story in a timely manner. There is a take > away for all of us, take it for what its worth. > > The weekend is here, time to recharge, empty your head. Like Karl says, > remember to clean up after yourselves and turn off the light when you're > done. > > I almost forgot, this link is posted off http://wiki.outages.org -> > Dashboard -> Lesson Learned What link would that be? > > > Enjoy:) > > > regards, > /virendra > > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iD8DBQFLm7ntpbZvCIJx1bcRAnBZAJ9bbfjbUwsvxvNlZemYG8unVSjPNwCfSxu/ > XMKz4by6UwE67fVWv4Ecp3k= > =hCAC > -----END PGP SIGNATURE----- > _______________________________________________ > Outages-discussion mailing list > Outages-discussion at outages.org > https://puck.nether.net/mailman/listinfo/outages-discussion > -- Democracy: Three wolves and a sheep voting on the dinner menu. (A republic, using parliamentary law, protects the minority.) Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml From virendra.rode at gmail.com Sat Mar 13 12:48:43 2010 From: virendra.rode at gmail.com (virendra rode) Date: Sat, 13 Mar 2010 09:48:43 -0800 Subject: [Outages-discussion] Interesting Read - Post-mortem for GoogleApp Engine on February 24th, 2010 outage In-Reply-To: <4B9BCF02.8090600@cox.net> References: <4B9BB9ED.2030702@gmail.com> <4B9BCF02.8090600@cox.net> Message-ID: <4B9BCFFB.50904@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Larry Sheldon wrote: > On 3/13/2010 10:14, virendra rode wrote: > Hi, > > Headless chicken syndrome that took place during this outage, kinda what > we all experience in our own environment. > > Yes there is lesson to be learned from this. Update your process and > keep your management network robust so your health status dashboard is > reachable to enable the whole story in a timely manner. There is a take > away for all of us, take it for what its worth. > > The weekend is here, time to recharge, empty your head. Like Karl says, > remember to clean up after yourselves and turn off the light when you're > done. > > I almost forgot, this link is posted off http://wiki.outages.org -> > Dashboard -> Lesson Learned > >> What link would that be? - ------------------------------------------- https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf?pli=1 regards, /virendra > > Enjoy:) > > > regards, > /virendra > > > > _______________________________________________ Outages-discussion mailing list Outages-discussion at outages.org https://puck.nether.net/mailman/listinfo/outages-discussion >> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFLm8/7pbZvCIJx1bcRAm9QAJwJ981RW8RuGSuBA1fq9RmTo9iv3ACfX4+N NRo1dufDtKHejn+g/CQ3XMg= =gTg8 -----END PGP SIGNATURE----- From mloftis at wgops.com Sat Mar 13 13:07:50 2010 From: mloftis at wgops.com (Michael Loftis) Date: Sat, 13 Mar 2010 11:07:50 -0700 Subject: [Outages-discussion] Interesting Read - Post-mortem for GoogleApp Engine on February 24th, 2010 outage In-Reply-To: <4B9BCF02.8090600@cox.net> References: <4B9BB9ED.2030702@gmail.com> <4B9BCF02.8090600@cox.net> Message-ID: <1A678A64B20FBE08280FD63A@[192.168.1.44]> --On Saturday, March 13, 2010 11:44 AM -0600 Larry Sheldon wrote: -> Lesson Learned > > What link would that be? The dashboard link is non-obvious and buried in the home page text. The dashboard is at The actual article is at: From virendra.rode at gmail.com Sat Mar 13 17:54:00 2010 From: virendra.rode at gmail.com (virendra rode) Date: Sat, 13 Mar 2010 14:54:00 -0800 Subject: [Outages-discussion] Interesting Read - Post-mortem for GoogleApp Engine on February 24th, 2010 outage In-Reply-To: <1A678A64B20FBE08280FD63A@[192.168.1.44]> References: <4B9BB9ED.2030702@gmail.com> <4B9BCF02.8090600@cox.net> <1A678A64B20FBE08280FD63A@[192.168.1.44]> Message-ID: <4B9C1788.5040603@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael, I agree. I just BOLD 'dashboard' hopefully it sticks out among other text. regards, /virendra Michael Loftis wrote: > > > --On Saturday, March 13, 2010 11:44 AM -0600 Larry Sheldon > wrote: > > -> Lesson Learned >> >> What link would that be? > > The dashboard link is non-obvious and buried in the home page text. The > dashboard is at > > The actual article is at: > > > > _______________________________________________ > Outages-discussion mailing list > Outages-discussion at outages.org > https://puck.nether.net/mailman/listinfo/outages-discussion > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFLnBeIpbZvCIJx1bcRAgkdAJ43Doi8rie1v9VvuwmTDSnpkGlBIgCeLdoU JuZp1wIo5iAaXzbxhaTjSgk= =YF7F -----END PGP SIGNATURE----- From cmadams at hiwaay.net Wed Mar 24 21:07:27 2010 From: cmadams at hiwaay.net (Chris Adams) Date: Wed, 24 Mar 2010 20:07:27 -0500 Subject: [Outages-discussion] Wikipedia suffers global outage. In-Reply-To: <4BAA82ED.5050309@gmail.com> References: <4BAA82ED.5050309@gmail.com> Message-ID: <20100325010727.GB1371912@hiwaay.net> > Update: Unfortunately, for many, this outage seems to have lasted longer > than an hour. It appears that many ISPs? DNS resolvers do not honor the > so-called Negative Cache TTL that we send (1 hour), and instead use a > longer value. We have circumvented this problem by renaming the affected > DNS record to something else. I'm curious: what software/settings are these "many ISPs" using that does this? I've seen this mentioned before, but BIND for example doesn't have an option to do this IIRC. -- Chris Adams Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. From list-outages at dragon.net Wed Mar 24 23:25:33 2010 From: list-outages at dragon.net (paul e) Date: Wed, 24 Mar 2010 20:25:33 -0700 Subject: [Outages-discussion] Wikipedia suffers global outage. In-Reply-To: <20100325010727.GB1371912@hiwaay.net> References: <4BAA82ED.5050309@gmail.com> <20100325010727.GB1371912@hiwaay.net> Message-ID: <20100325032533.6821BC3DC21@scatha.remote.dragon.net> >> Update: Unfortunately, for many, this outage seems to have lasted >> longer than an hour. It appears that many ISPs=92 DNS resolvers do >> not honor the so-called Negative Cache TTL that we send (1 hour), and >> instead use a longer value. We have circumvented this problem by >> renaming the affected DNS record to something else. cmadams> I'm curious: what software/settings are these "many ISPs" using cmadams> that does this? I've seen this mentioned before, but BIND for cmadams> example doesn't have an option to do this IIRC. ncache is set on the auth server for the zone, in the SOA record. It's the 'minimum' timer, the last of the 4 timers after serial number. See RFC 2308 for how negative caching works. Any RFC compliant resolver should deal with this correctly. BIND does the correct thing, both on the auth server side and as a recursive resolver.