[Outages-discussion] DNS server caused provincial-wide outage

[Joe Abley]

On 26 May 2015, at 15:34, Frank Bulk wrote:

> http://www.620ckrm.com/ckrm-on-air/ckrm-local-news/10016-sasktel-experiencin
> g-major-internet-outage-across-saskatchewan
>
> SaskTel had a major internet outage across the province.
>
> Spokesperson Michelle Englot says the problem lasted a couple of hours 
> and
> was repaired early in the evening.
>
> A DNS server failure, which is the company's main connection to the 
> internet
> was the reason for the crash as it malfunctioned for some reason
> interrupting wireless and landline connections.
>
> Not very clear.  Must be more going on.

Correlates well with other "nation! wide! internet! outages!" reported 
by the national press in Canada in recent years which, with a small 
amount of googling, reveals workarounds such as "change your DNS setting 
to 8.8.8.8".

People (perhaps especially Telcos) continue to believe that "carrier 
grade" when applied to DNS resolution means you must have layers of 
load-balancers and firewalls in front of the machines answering the 
queries, because that's how "carrier grade" is achieved.

Out here in the real world, we know that any attempt to keep state on 
the path between clients and servers of a stateless protocol, especially 
one with lots of clients where transactions are very short-lived, is a 
recipe for flames and carnage.

I know nothing at all about what actually happened at SaskTel. But I'll 
bet you $5 I'm on the right track.


Joe