[cisco-voip] Webex Teams Down

Tue Sep 25 11:47:58 EDT 2018

I don't know why, but I like the beginning of this sentence:

"However, once this thermal buffer was depleted..."

I'm going to use that in my next RCA for a Call Center outage.

On Tue, Sep 25, 2018 at 10:43 AM Matthew Loraditch <
MLoraditch at heliontechnologies.com> wrote:

> This one almost seems like an "act of god" or hacking had to take place.
> If you want to read about an interesting one MS had with Azure a few weeks
> ago look here on 9/4 https://azure.microsoft.com/en-us/status/history/
> <https://azure.microsoft.com/en-us/status/history/>
> Azure status history | Microsoft Azure
> <https://azure.microsoft.com/en-us/status/history/>
> Check the status history of Microsoft Azure services here.
> azure.microsoft.com
>
>
> Matthew Loraditch
> Sr. Network Engineer
> p: *443.541.1518* <443.541.1518>
> w: *www.heliontechnologies.com* <http://www.heliontechnologies.com/>  |
> e: *MLoraditch at heliontechnologies.com* <MLoraditch at heliontechnologies.com>
> [image: Facebook] <https://facebook.com/heliontech>
> [image: Twitter] <https://twitter.com/heliontech>
> [image: LinkedIn] <https://www.linkedin.com/company/helion-technologies>
> ------------------------------
> *From:* cisco-voip <cisco-voip-bounces at puck.nether.net> on behalf of Ryan
> Huff <ryanhuff at outlook.com>
> *Sent:* Tuesday, September 25, 2018 11:37 AM
> *To:* Lelio Fulgenzi
> *Cc:* Cisco VoIP Group
> *Subject:* Re: [cisco-voip] Webex Teams Down
>
> That’s the value prop of the cloud though; “we take away the risk of your
> on-prem infrastructure, but you have to pay for that in control of your
> service, experience and ability to troubleshoot”.
>
> I tell folks all the time risk mitigation paid for by giving up control is
> never a good idea.
>
> - Ryan
>
> On Sep 25, 2018, at 11:30, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:
>
>
>
> I know that for my maintenance activities I evaluate the risk vs potential
> impact vs backout requirements to understand how much effort I put in to
> preparing things. One time I scavenged our e-waste pile for compatible HP
> hardware and purchased old CPUs for an upgrade to ensure that I had at most
> a 90minute backout (plugging wires and turning up servers x 9). Restoring
> from backup was just too long in this case.
>
>
>
> I don’t begin to fathom the complexity of the webex backend service, but I
> find it hard to believe there were not options available to ensure a quick
> and orderly backout plan – not simply restoring from backup. Would it be
> expensive? Sure, but define “expensive” relative to the impact this has
> caused.
>
>
>
>
>
>
>
>
>
>
>
>
>
> ---
>
> *Lelio Fulgenzi, B.A.* | Senior Analyst
>
> Computing and Communications Services | University of Guelph
>
> Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON |
> N1G 2W1
>
> 519-824-4120 Ext. 56354 | lelio at uoguelph.ca
>
>
>
> www.uoguelph.ca/ccs
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.uoguelph.ca%2Fccs&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=wmNZlQ1OIZdMxkQo%2FZR%2BpvyIG0uGwIJPIUPkyTMzgM4%3D&reserved=0>
> | @UofGCCS on Instagram, Twitter and Facebook
>
>
>
> <image001.png>
>
>
>
> *From:* cisco-voip <cisco-voip-bounces at puck.nether.net> *On Behalf Of *Ryan
> Huff
> *Sent:* Tuesday, September 25, 2018 11:20 AM
> *To:* Anthony Holloway <avholloway+cisco-voip at gmail.com>
> *Cc:* Cisco VoIP Group <cisco-voip at puck.nether.net>
> *Subject:* Re: [cisco-voip] Webex Teams Down
>
>
>
> I my 17 years doing this, I have never once caused or suffered a seemingly
> unexplained and unexpected 10+ hour outage of all service to an
> environment/application.
>
>
>
> That’s the difference, control.
>
>
>
> Sent from my iPhone
>
>
> On Sep 25, 2018, at 11:05, Anthony Holloway <
> avholloway+cisco-voip at gmail.com> wrote:
>
> In my mind, this is no different than an on prem
> migration/upgrade/change/etc going south.  The fact that the impacted
> service is in the cloud, is not important.  Even if the Internet connection
> itself went down, that's no different than a private MPLS outage.
>
>
>
> From an end user's perspective, whether cloud or on prem, the service is
> simply unavailable.  From a CIO perspective, someone else screwed up, and
> someone else is working on it.  I don't see why this is different or
> interesting.
>
>
>
> What is it that do I not get?
>
>
>
> On Tue, Sep 25, 2018 at 9:11 AM Palmer, Brian <Brian.Palmer at bcbsfl.com>
> wrote:
>
> This sure is rather wild that it has been down for so long.  We only have
> a few boards, but I do hope that every business user is reviewing the word
> “cloud” and how great it really is within this context.
>
>
>
>
>
> *From:* cisco-voip <cisco-voip-bounces at puck.nether.net> *On Behalf Of *Matthew
> Loraditch
> *Sent:* Tuesday, September 25, 2018 9:43 AM
> *To:* Ryan Huff <ryanhuff at outlook.com>; Matt Jacobson <
> m4ttjacobson at gmail.com>
> *Cc:* cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Webex Teams Down
>
>
>
> The RFO for this one is going to be a doozy. On meetings they are
> migrating all users who had been upgraded to Video Platform 2.0 back to
> 1.0. I have no idea of the scope but that seems like a pretty big
> undertaking and a tiny insight into how screwed up whatever happened is.
>
>
>
>
>
> *Matthew Loraditch***
>
> *Sr. Network Engineer*
>
> p: *443.541.1518* <443.541.1518>
>
> w: *http://defang.bcbsfl.com/defang.php?url=www.heliontechnologies.com*
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttp%3A%2F%2Fwww.heliontechnologies.com%2F&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=ErnApJU4sVTVGvv4VOZS%2BfIgi4NgwQrjgKi4C1zA7%2F8%3D&reserved=0>
>
>  |
>
> e: *MLoraditch at heliontechnologies.com* <MLoraditch at heliontechnologies.com>
>
> [image: Facebook]
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Ffacebook.com%2Fheliontech&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=hTymIht3fd6eTaa3cazbZtFsoIz7TiVVvdbnktGrOl8%3D&reserved=0>
>
> [image: Twitter]
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fheliontech&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=PKWIWg7BUBTL3tCoLnN5LTGLF0%2F0wwFccKqTlVtBFRk%3D&reserved=0>
>
> [image: LinkedIn]
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fhelion-technologies&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=EYkE%2BbBosqsvfCHGtDW%2BlVCLGyE9BRJcC2N8jBvy6Rc%3D&reserved=0>
>
> *From:* cisco-voip <cisco-voip-bounces at puck.nether.net> *On Behalf Of *Ryan
> Huff
> *Sent:* Tuesday, September 25, 2018 9:36 AM
> *To:* Matt Jacobson <m4ttjacobson at gmail.com>
> *Cc:* cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Webex Teams Down
>
>
>
> Who wants to go in with me on creating an on-prem DR application for
> individual WebEx Teams domains? Lol ... the cloud ... yup, let’s do that
> ...
>
>
>
> The cloud is a little like networking before segmentation .... anything
> impacts everything 🤔
>
> The only things I can think of that could possibly take this long would be
> like having to restore things from backup or waiting on uber large database
> indices to rebuild or reindex ..... hope things get restored soon!
>
>
> On Sep 25, 2018, at 07:22, Matt Jacobson <m4ttjacobson at gmail.com> wrote:
>
> Webex meetings is still working, but you can’t dial in with video
> endpoints.
>
>
>
> On Tue, Sep 25, 2018 at 15:10 Bernhard Albler <bernhard.albler at gmail.com>
> wrote:
>
> yeah, crazy outage, basically the full working day for us here in Europe
>
> it also impacts CMR for customers on Video Platform v2 ( which should be
> most)
>
>
>
> On Tue 25. Sep 2018 at 12:39, Ryan Huff <ryanhuff at outlook.com> wrote:
>
> I haven't seen one this long this since the early days of Spark ... been
> down since 9:22PM eastern on 9-24-18 ... wow ....
>
>
>
>  <image1.png>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
>
> http://defang.bcbsfl.com/defang.php?url=https://puck.nether.net/mailman/listinfo/cisco-voip
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fpuck.nether.net%252Fmailman%252Flistinfo%252Fcisco-voip%26data%3D02%257C01%257C%257Cd2215ff5dbe84dc2584b08d622d93f0b%257C84df9e7fe9f640afb435aaaaaaaaaaaa%257C1%257C0%257C636734713787409869%26sdata%3D2nNGVvI7Vhw%252BCwkk67YzIUpauo422OnoXc1A6TeYyxo%253D%26reserved%3D0&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=i6qroflrPcduCG3dHgQ66VOYrR69ScSOLqdA0ndaz0c%3D&reserved=0>
>
> --
>
> Bernhard Albler, +4369917207384
> --
> "Was Nachwelt! Wie komm' ich dazu was für die Nachwelt zu tun? Was hat
> denn die Nachwelt für mich getan?"
> --Carl Friedrich Zelter
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
>
> http://defang.bcbsfl.com/defang.php?url=https://puck.nether.net/mailman/listinfo/cisco-voip
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttps%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fpuck.nether.net%252Fmailman%252Flistinfo%252Fcisco-voip%26data%3D02%257C01%257C%257Cd2215ff5dbe84dc2584b08d622d93f0b%257C84df9e7fe9f640afb435aaaaaaaaaaaa%257C1%257C0%257C636734713787409869%26sdata%3D2nNGVvI7Vhw%252BCwkk67YzIUpauo422OnoXc1A6TeYyxo%253D%26reserved%3D0&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=i6qroflrPcduCG3dHgQ66VOYrR69ScSOLqdA0ndaz0c%3D&reserved=0>
>
> We comply with applicable Federal civil rights laws and do not
> discriminate.
>
> You may access the Non-Discrimination and Accessibility Notice here
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffloridablue.com%2Fndnotice&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=MNW1BH85AWtatzYX7B8YOvyWAMeL1pqx%2B2B5iPmgTho%3D&reserved=0>
> .
>
> Language Assistance Available:
>
> Español, Kreyol Ayisien, Tiếng Việt, Português, 中文, français, Tagalog,
> русский, italiano, Deutsche, 한국어, Polskie, Gujarati, ไทย, العربية, 日本語,
> فارسی
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffloridablue.com%2Flanguageservices&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862088992900&sdata=hdkRq4n8Qx%2BFTeLtuTaDdGnZZJ6i51VopabbsB5pgV0%3D&reserved=0>
>
> Florida Blue is a trade name of Blue Cross and Blue Shield of Florida,
> Inc.  Blue Cross and Blue Shield of Florida, Inc., and its subsidiary and
> affiliate companies are not responsible for errors or omissions in this
> e-mail message. Any personal comments made in this e-mail do not reflect
> the views of Blue Cross and Blue Shield of Florida, Inc.  The information
> contained in this document may be confidential and intended solely for the
> use of the individual or entity to whom it is addressed.  This document may
> contain material that is privileged or protected from disclosure under
> applicable law.  If you are not the intended recipient or the individual
> responsible for delivering to the intended recipient, please (1) be advised
> that any use, dissemination, forwarding, or copying of this document IS
> STRICTLY PROHIBITED; and (2) notify sender immediately by telephone and
> destroy the document. THANK YOU.
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862089149153&sdata=FSi2q%2Bs6m6IKpxiBAEAoLmMNcHcBcWCBkoxfnu1wBcM%3D&reserved=0>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7Ca10d1791a9b34c9b0c2408d622f86156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734847508091071&sdata=uVYGNqnf6KI5c6AjBnCjdSwHi5pdBzvsg5KQoM0Qn4I%3D&reserved=0
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7C2402c8f43c8c4758269808d622fbc66c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636734862089149153&sdata=FSi2q%2Bs6m6IKpxiBAEAoLmMNcHcBcWCBkoxfnu1wBcM%3D&reserved=0>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/e3564567/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image098391.png
Type: image/png
Size: 8404 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/e3564567/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image189261.png
Type: image/png
Size: 431 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/e3564567/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image606166.png
Type: image/png
Size: 561 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/e3564567/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image395940.png
Type: image/png
Size: 444 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/e3564567/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image395940.png
Type: image/png
Size: 444 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20180925/e3564567/attachment-0004.png>