[cisco-voip] Webex Teams Down

Tue Sep 25 11:30:06 EDT 2018

I know that for my maintenance activities I evaluate the risk vs potential impact vs backout requirements to understand how much effort I put in to preparing things. One time I scavenged our e-waste pile for compatible HP hardware and purchased old CPUs for an upgrade to ensure that I had at most a 90minute backout (plugging wires and turning up servers x 9). Restoring from backup was just too long in this case.

I don’t begin to fathom the complexity of the webex backend service, but I find it hard to believe there were not options available to ensure a quick and orderly backout plan – not simply restoring from backup. Would it be expensive? Sure, but define “expensive” relative to the impact this has caused.

---
Lelio Fulgenzi, B.A. | Senior Analyst
Computing and Communications Services | University of Guelph
Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON | N1G 2W1
519-824-4120 Ext. 56354 | lelio at uoguelph.ca<mailto:lelio at uoguelph.ca>

www.uoguelph.ca/ccs<http://www.uoguelph.ca/ccs> | @UofGCCS on Instagram, Twitter and Facebook

[University of Guelph Cornerstone with Improve Life tagline]

From: cisco-voip <cisco-voip-bounces at puck.nether.net> On Behalf Of Ryan Huff
Sent: Tuesday, September 25, 2018 11:20 AM
To: Anthony Holloway <avholloway+cisco-voip at gmail.com>
Cc: Cisco VoIP Group <cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] Webex Teams Down

I my 17 years doing this, I have never once caused or suffered a seemingly unexplained and unexpected 10+ hour outage of all service to an environment/application.

That’s the difference, control.

Sent from my iPhone

On Sep 25, 2018, at 11:05, Anthony Holloway <avholloway+cisco-voip at gmail.com<mailto:avholloway+cisco-voip at gmail.com>> wrote:
In my mind, this is no different than an on prem migration/upgrade/change/etc going south.  The fact that the impacted service is in the cloud, is not important.  Even if the Internet connection itself went down, that's no different than a private MPLS outage.

From an end user's perspective, whether cloud or on prem, the service is simply unavailable.  From a CIO perspective, someone else screwed up, and someone else is working on it.  I don't see why this is different or interesting.

What is it that do I not get?

On Tue, Sep 25, 2018 at 9:11 AM Palmer, Brian <Brian.Palmer at bcbsfl.com<mailto:Brian.Palmer at bcbsfl.com>> wrote:
This sure is rather wild that it has been down for so long.  We only have a few boards, but I do hope that every business user is reviewing the word “cloud” and how great it really is within this context.

From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> On Behalf Of Matthew Loraditch
Sent: Tuesday, September 25, 2018 9:43 AM
To: Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>>; Matt Jacobson <m4ttjacobson at gmail.com<mailto:m4ttjacobson at gmail.com>>
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] Webex Teams Down

The RFO for this one is going to be a doozy. On meetings they are migrating all users who had been upgraded to Video Platform 2.0 back to 1.0. I have no idea of the scope but that seems like a pretty big undertaking and a tiny insight into how screwed up whatever happened is.

Matthew Loraditch

Sr. Network Engineer

p: 443.541.1518<tel:443.541.1518>

w: http://defang.bcbsfl.com/defang.php?url=www.heliontechnologies.com<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttp%3A%2F%2Fwww.heliontechnologies.com%2F&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=uXL7JSJPsyEqfh0fIe5bhiCd5WEmW1bwIFBkQ8INuzE%3D&reserved=0>

 |

e: MLoraditch at heliontechnologies.com<mailto:MLoraditch at heliontechnologies.com>

[cid:image001.png at 01D454B8.19DE9440]

[Facebook]<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Ffacebook.com%2Fheliontech&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=1FsLL1I5tlE18UfoHFAghLVkvL%2B0rx5BtjMcX69hNio%3D&reserved=0>

[Twitter]<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fheliontech&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=75BhR77OcbhxO9VttwTqWUCG6NGYPs7MCI9NdHbDFKQ%3D&reserved=0>

[LinkedIn]<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fhelion-technologies&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=rH9O9Mlp1AEDp9Amd1U52g58MvW%2B%2FMapbe3zZZfkIDE%3D&reserved=0>

From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> On Behalf Of Ryan Huff
Sent: Tuesday, September 25, 2018 9:36 AM
To: Matt Jacobson <m4ttjacobson at gmail.com<mailto:m4ttjacobson at gmail.com>>
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] Webex Teams Down

Who wants to go in with me on creating an on-prem DR application for individual WebEx Teams domains? Lol ... the cloud ... yup, let’s do that ...

The cloud is a little like networking before segmentation .... anything impacts everything 🤔
The only things I can think of that could possibly take this long would be like having to restore things from backup or waiting on uber large database indices to rebuild or reindex ..... hope things get restored soon!

On Sep 25, 2018, at 07:22, Matt Jacobson <m4ttjacobson at gmail.com<mailto:m4ttjacobson at gmail.com>> wrote:
Webex meetings is still working, but you can’t dial in with video endpoints.

On Tue, Sep 25, 2018 at 15:10 Bernhard Albler <bernhard.albler at gmail.com<mailto:bernhard.albler at gmail.com>> wrote:
yeah, crazy outage, basically the full working day for us here in Europe
it also impacts CMR for customers on Video Platform v2 ( which should be most)

On Tue 25. Sep 2018 at 12:39, Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
I haven't seen one this long this since the early days of Spark ... been down since 9:22PM eastern on 9-24-18 ... wow ....

 <image1.png>
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
http://defang.bcbsfl.com/defang.php?url=https://puck.nether.net/mailman/listinfo/cisco-voip<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fpuck.nether.net%252Fmailman%252Flistinfo%252Fcisco-voip%26data%3D02%257C01%257C%257Cd2215ff5dbe84dc2584b08d622d93f0b%257C84df9e7fe9f640afb435aaaaaaaaaaaa%257C1%257C0%257C636734713787409869%26sdata%3D2nNGVvI7Vhw%252BCwkk67YzIUpauo422OnoXc1A6TeYyxo%253D%26reserved%3D0&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=6HCBu7P%2FWQ%2FRyqfa9%2BFudZ1rqTI0%2Bs1WYBoJUSspUbg%3D&reserved=0>
--
Bernhard Albler, +4369917207384
--
"Was Nachwelt! Wie komm' ich dazu was für die Nachwelt zu tun? Was hat denn die Nachwelt für mich getan?"
--Carl Friedrich Zelter
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
http://defang.bcbsfl.com/defang.php?url=https://puck.nether.net/mailman/listinfo/cisco-voip<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fpuck.nether.net%252Fmailman%252Flistinfo%252Fcisco-voip%26data%3D02%257C01%257C%257Cd2215ff5dbe84dc2584b08d622d93f0b%257C84df9e7fe9f640afb435aaaaaaaaaaaa%257C1%257C0%257C636734713787409869%26sdata%3D2nNGVvI7Vhw%252BCwkk67YzIUpauo422OnoXc1A6TeYyxo%253D%26reserved%3D0&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=6HCBu7P%2FWQ%2FRyqfa9%2BFudZ1rqTI0%2Bs1WYBoJUSspUbg%3D&reserved=0>

We comply with applicable Federal civil rights laws and do not discriminate.

You may access the Non-Discrimination and Accessibility Notice here<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffloridablue.com%2Fndnotice&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=fFATlhi%2BtkeTuNWAlMpD%2BHOeEpOpewgP%2BcsLIeSkizg%3D&reserved=0>.

Language Assistance Available:

Español, Kreyol Ayisien, Tiếng Việt, Português, 中文, français, Tagalog, русский, italiano, Deutsche, 한국어, Polskie, Gujarati, ไทย, العربية, 日本語, فارسی<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffloridablue.com%2Flanguageservices&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=3%2Bf1GT19tF1kmBPAv%2Bmq4AyzDbRVHhJgOswXXg4iZww%3D&reserved=0>

Florida Blue is a trade name of Blue Cross and Blue Shield of Florida, Inc.  Blue Cross and Blue Shield of Florida, Inc., and its subsidiary and affiliate companies are not responsible for errors or omissions in this e-mail message. Any personal comments made in this e-mail do not reflect the views of Blue Cross and Blue Shield of Florida, Inc.  The information contained in this document may be confidential and intended solely for the use of the individual or entity to whom it is addressed.  This document may contain material that is privileged or protected from disclosure under applicable law.  If you are not the intended recipient or the individual responsible for delivering to the intended recipient, please (1) be advised that any use, dissemination, forwarding, or copying of this document IS STRICTLY PROHIBITED; and (2) notify sender immediately by telephone and destroy the document. THANK YOU.
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847507934813&sdata=ZVHfaPRlfj973tGeuJZwTwPwoMb4ow0Yr1gOtanG1Ac%3D&reserved=0>
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847508091071&sdata=uVYGNqnf6KI5c6AjBnCjdSwHi5pdBzvsg5KQoM0Qn4I%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/06db4abe/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1297 bytes
Desc: image001.png
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/06db4abe/attachment.png>