<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'><font face="arial, helvetica, sans-serif"><span style="font-size: 13.3333px;">Our investigation determined that a router in our Ashburn data center where the STP is located was incorrectly learning the route back to our Ashburn VPN site via the IPX core in Chicago ( though Chicago was not actually in the path). The power issue in Chicago that took down an IPX switch, impacted the path from our Dallas VPN site to the Chicago STP as expected, but the loss of the route to Ashburn was not expected. TNS has corrected the Ashburn routing as an emergency break-fix (non-impacting). TNS engineers have confirmed that the router in Ashburn is now learning the route from our IPX core in Ashburn as designed. Please let us know if you have any questions. </span></font><br><br><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><span name="x"></span><br><br>-----<br>Mike Hammett<br>Intelligent Computing Solutions<br>http://www.ics-il.com<br><br><br><br>Midwest Internet Exchange<br>http://www.midwest-ix.com<br><br><span name="x"></span><br></div><br><hr id="zwchr" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><div style="color: rgb(0, 0, 0); font-family: Helvetica, Arial, sans-serif; font-size: 12pt; font-weight: normal; font-style: normal; text-decoration: none;"><b>From: </b>"Mary Lou Carey" <marylou@backuptelecom.com><br><b>To: </b>"Mike Hammett" <voiceops@ics-il.net><br><b>Cc: </b>"voiceops" <voiceops@voiceops.org><br><b>Sent: </b>Thursday, January 20, 2022 6:23:28 PM<br><b>Subject: </b>Re: [VoiceOps] TNS SS7 Outage<br><br>That sounds as believable as the Christmas Day bombing in Nashville that <br>took out local and 911 phone services out all the way up to St Louis. <br>Anyone who's familiar with the network knows St Louis isn't in the same <br>LATA and a building built to withstand natural disasters etc wouldn't be <br>damaged that badly by a bomb going off down the street from the <br>building.<br><br>You'd better be asking for a link to a different set of STPs because if <br>the power in Chicago took out all 4 of their STPs, they have some <br>serious problems with their network redundancy plan!<br><br>MARY LOU CAREY<br>BackUP Telecom Consulting<br>Office: 615-791-9969<br>Cell: 615-796-1111<br><br>On 2022-01-20 05:44 AM, Mike Hammett wrote:<br>> Currently the story is that there was a power supply issue in Chicago.<br>> <br>> Okay, so Chicago went down. What about Ashburn? Just how big was that<br>> power supply? ;-)<br>> <br>> I know TNS has more facilities, so I'm going to try to get another<br>> link out of them gratis.<br>> <br>> -----<br>> Mike Hammett<br>> Intelligent Computing Solutions<br>> http://www.ics-il.com<br>> <br>> Midwest Internet Exchange<br>> http://www.midwest-ix.com<br>> <br>> -------------------------<br>> <br>> From: "Mike Hammett" <voiceops@ics-il.net><br>> To: "voiceops" <voiceops@voiceops.org><br>> Sent: Wednesday, January 19, 2022 4:13:08 PM<br>> Subject: Re: [VoiceOps] TNS SS7 Outage<br>> <br>> The links have since come back. This is the second major failure of<br>> TNS within 6 months. No RFO yet, but I will certainly be on them for<br>> some detail.<br>> <br>> My Metaswitch support rep verified that (pardon my lack of using the<br>> correct terminology, I'm not that familiar with it) our SS7 links<br>> never actually went down, it just lost connectivity beyond that. Sorta<br>> like my BGP peer stayed up, but I lost the global routing table behind<br>> it.<br>> <br>> -----<br>> Mike Hammett<br>> Intelligent Computing Solutions<br>> http://www.ics-il.com<br>> <br>> Midwest Internet Exchange<br>> http://www.midwest-ix.com<br>> <br>> -------------------------<br>> <br>> From: "Mike Hammett" <voiceops@ics-il.net><br>> To: "voiceops" <voiceops@voiceops.org><br>> Sent: Wednesday, January 19, 2022 4:05:32 PM<br>> Subject: [VoiceOps] TNS SS7 Outage<br>> <br>> Does anyone have any information on the TNS SS7 outage going on now?<br>> Both of our A-links go to different facilities (Chicago and Ashburn)<br>> over different VPNs (which are up).<br>> <br>> TNS has acknowledged that it's a carrier issue. That they're<br>> investigating.<br>> <br>> -----<br>> Mike Hammett<br>> Intelligent Computing Solutions<br>> http://www.ics-il.com<br>> <br>> Midwest Internet Exchange<br>> http://www.midwest-ix.com<br>> <br>> _______________________________________________<br>> VoiceOps mailing list<br>> VoiceOps@voiceops.org<br>> https://puck.nether.net/mailman/listinfo/voiceops<br>> <br>> _______________________________________________<br>> VoiceOps mailing list<br>> VoiceOps@voiceops.org<br>> https://puck.nether.net/mailman/listinfo/voiceops<br>> _______________________________________________<br>> VoiceOps mailing list<br>> VoiceOps@voiceops.org<br>> https://puck.nether.net/mailman/listinfo/voiceops<br></div><br></div></body></html>