[Outages-discussion] [outages] Anybody else having trouble with Centurylink toll-free #s?

[Frank Bulk]

It's hard to understand how a company that carries as many toll-free numbers as they do doesn't have systems in place that take hourly (or more) snapshots of those tables and provide mechanisms for the operators to roll back, plus that the script mentioned below don't have checks to limit the amount of damage (i.e. if it's more than one row, warn the end-user before proceeding, and even then, don't allow more than 5 or 10 row deletions, make a backup before proceeding, etc.).

Frank

-----Original Message-----
From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Brad Fleming via Outages
Sent: Thursday, September 03, 2015 6:28 PM
To: Sarlas, George <gsarlas at irhythmtech.com>
Cc: outages at outages.org
Subject: Re: [outages] Anybody else having trouble with Centurylink toll-free #s?

I don’t guarantee the validity of the data and we don’t have any numbers impacted (no voice services with CL).
https://www.reddit.com/r/sysadmin/comments/3jjdgs/centurylink_1800_outage_pacific_northwest/

Of interest is this:
5:46pm central:  What Happened
Data structure error were observed on the 8xx data table.   A script was  run to correct the errors, and wiped out the 8xx data table which  caused every CenturyLink provided 8xx numbers to stop processing calls.   Cause is unknown at this time, isolation and investigations are  ongoing.  Trouble identified as an 8xx database issue.  Next Steps
Most current backup database has been identified and is being reloaded  currently.  This reload is expected to be completed in approximately 20  minutes if no issues are encountered.  All critical teams are engaged in  restoration efforts.  edit 2:   from Centurylink at 5:16pm central:   REPORT OF MAJOR SERVICE FAILURE OR OCCURRENCE TICKET # CO.090315.009/PURPLE/UPDATED
Network(s)      National
Location(s)     DENVER/CO
NIL Level; Status       PURPLE; UPDATED (U)
Failure Category        VOICE
Action Taken    UPGRADED TO PURPLE DUE TO WIDESPREAD 800 OUTAGE THROUGHOUT THE COUNTRY.
Services Affected/Impacts       TOLL FREE SERVICES 800;866;888(Voice Customer Count 100)


Again, this is at best second-hand data.


> On Sep 3, 2015, at 4:16 PM, Sarlas, George via Outages <outages at outages.org> wrote:
> 
>  
> 
>  
> 
> ----
> 
> George Sarlas
> 
> Sr. Manager, IT Operations
> 
> iRhythm Technologies, Inc.
> 
>  
> 
> 2 Marriott Dr.
> 
> Lincolnshire, IL 60069
> 
>  
> 
> email: gsarlas at irhythmtech.com
> 
> phone: 224-543-4253
> 
>  
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages