[outages] Long latency in receiving email on this list/Other observations about today's Google Incident

[Bob Roswell]

I am looking through the emails from the outages list from today's
Google incident and many of the posts have original time stamps.  I
received my copies of mailing lists posts over an hour after they were
sent.  For example, I posted a message today at 12:11PM EDT, and I
received my copy from the list at 1:20PM.  We were well into our
troubleshooting with Verizon and Level 3 when I saw the first post on
this list.  Is there anything we can do to speed the delivery of the
emails from this list?   We have had a better incident response had we
known sooner that this was a global issue.  And I was still receiving
hour old trouble reports long after the problem was resolved.

 

I also called two out of three of our upstream providers (Level 3 and
Verizon)  Neither had status messages on their web sites, nor for their
dispatch folks.  We had all of the relevant information (account #,
Circuit ID etc) but they insisted on going through a long ticket process
(Have you verified power, what are your IPS, what IPs are you trying to
reach, what are your contact hours, what type of equipment do you have
etc) instead of just taking the call.  At Verizon, (50 Mbit circuit) I
was on hold for about 15 minutes before I was even able to place a
trouble ticket.

 

I also saw speculation on the list that it was a Level 3 problem, and in
fact we could not reach Google with any reliability on our DS3 with
Level3 (On level3 IP Space)  But our Comcast line uses Level 3 for
transport and we had no issues reaching Google services from Comcast.

 

Bob Roswell

System Source

broswell at syssrc.com

(410) 771-5544 ext 4336

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20090514/ea36ca85/attachment.htm>