[Outages-discussion] CenturyLink Outages this morning

[valdis.kletnieks at vt.edu]

On Thu, 27 Dec 2018 20:58:27 -0700, "Keith Medcalf" said:
> One wonders why they do not simply "undo" whatever change they made between
> when things were "working" and when they "broke" ...

Sometimes, it's not so simple.  One Boston medical school found out the hard way
in 2002:

"All of this has earned Halamka a considerable measure of renown. For two years
running, InformationWeek named Halamka's IT organization number one among
hospitals in its yearly ranking of innovative IT groups. In September 2002,
CareGroup was ranked 16th on InformationWeek's list of 500.

Two months later, Beth Israel Deaconess experienced one of the worst
health-care IT disasters ever. Over four days, Halamka's network crashed
repeatedly, forcing the hospital to revert to the paper patient-records system
that it had abandoned years ago. Lab reports that doctors normally had in hand
within 45 minutes took as long as five hours to process. The emergency
department diverted traffic for hours during the course of two days.
Ultimately, the hospital's network would have to be completely overhauled."

https://www.computerworld.com/article/2581420/disaster-recovery/all-systems-down.html

Some Cisco marketing stuff, but explains the gory details of how spanning tree failed.

https://community.cisco.com/kxiwq67737/attachments/kxiwq67737/6016-discussions-lan-switching-routing/239903/2/140889-CS401%20-%20All%20systems%20down%20-%20Edwin%20Hoffman%20CSA.pdf