[VoiceOps] TNS SS7 Outage

[Mike Hammett]

Our investigation determined that a router in our Ashburn data center where the STP is located was incorrectly learning the route back to our Ashburn VPN site via the IPX core in Chicago ( though Chicago was not actually in the path). The power issue in Chicago that took down an IPX switch, impacted the path from our Dallas VPN site to the Chicago STP as expected, but the loss of the route to Ashburn was not expected. TNS has corrected the Ashburn routing as an emergency break-fix (non-impacting). TNS engineers have confirmed that the router in Ashburn is now learning the route from our IPX core in Ashburn as designed. Please let us know if you have any questions. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 



----- Original Message -----

From: "Mary Lou Carey" <marylou at backuptelecom.com> 
To: "Mike Hammett" <voiceops at ics-il.net> 
Cc: "voiceops" <voiceops at voiceops.org> 
Sent: Thursday, January 20, 2022 6:23:28 PM 
Subject: Re: [VoiceOps] TNS SS7 Outage 

That sounds as believable as the Christmas Day bombing in Nashville that 
took out local and 911 phone services out all the way up to St Louis. 
Anyone who's familiar with the network knows St Louis isn't in the same 
LATA and a building built to withstand natural disasters etc wouldn't be 
damaged that badly by a bomb going off down the street from the 
building. 

You'd better be asking for a link to a different set of STPs because if 
the power in Chicago took out all 4 of their STPs, they have some 
serious problems with their network redundancy plan! 

MARY LOU CAREY 
BackUP Telecom Consulting 
Office: 615-791-9969 
Cell: 615-796-1111 

On 2022-01-20 05:44 AM, Mike Hammett wrote: 
> Currently the story is that there was a power supply issue in Chicago. 
> 
> Okay, so Chicago went down. What about Ashburn? Just how big was that 
> power supply? ;-) 
> 
> I know TNS has more facilities, so I'm going to try to get another 
> link out of them gratis. 
> 
> ----- 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> Midwest Internet Exchange 
> http://www.midwest-ix.com 
> 
> ------------------------- 
> 
> From: "Mike Hammett" <voiceops at ics-il.net> 
> To: "voiceops" <voiceops at voiceops.org> 
> Sent: Wednesday, January 19, 2022 4:13:08 PM 
> Subject: Re: [VoiceOps] TNS SS7 Outage 
> 
> The links have since come back. This is the second major failure of 
> TNS within 6 months. No RFO yet, but I will certainly be on them for 
> some detail. 
> 
> My Metaswitch support rep verified that (pardon my lack of using the 
> correct terminology, I'm not that familiar with it) our SS7 links 
> never actually went down, it just lost connectivity beyond that. Sorta 
> like my BGP peer stayed up, but I lost the global routing table behind 
> it. 
> 
> ----- 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> Midwest Internet Exchange 
> http://www.midwest-ix.com 
> 
> ------------------------- 
> 
> From: "Mike Hammett" <voiceops at ics-il.net> 
> To: "voiceops" <voiceops at voiceops.org> 
> Sent: Wednesday, January 19, 2022 4:05:32 PM 
> Subject: [VoiceOps] TNS SS7 Outage 
> 
> Does anyone have any information on the TNS SS7 outage going on now? 
> Both of our A-links go to different facilities (Chicago and Ashburn) 
> over different VPNs (which are up). 
> 
> TNS has acknowledged that it's a carrier issue. That they're 
> investigating. 
> 
> ----- 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> Midwest Internet Exchange 
> http://www.midwest-ix.com 
> 
> _______________________________________________ 
> VoiceOps mailing list 
> VoiceOps at voiceops.org 
> https://puck.nether.net/mailman/listinfo/voiceops 
> 
> _______________________________________________ 
> VoiceOps mailing list 
> VoiceOps at voiceops.org 
> https://puck.nether.net/mailman/listinfo/voiceops 
> _______________________________________________ 
> VoiceOps mailing list 
> VoiceOps at voiceops.org 
> https://puck.nether.net/mailman/listinfo/voiceops 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20220122/95123629/attachment-0001.htm>