<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:Consolas;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
p.imprintuniqueid, li.imprintuniqueid, div.imprintuniqueid
{mso-style-name:imprintuniqueid;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Just received this afternoon:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>==================<o:p></o:p></p><p><span style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>Support Team,<o:p></o:p></span></p><p><span style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>You have several business customers being affected by an ongoing issue. In order to resolve this, First Data is requesting that you clear the cache on all DNS servers being used to support them. We propagated a correction over 16 hours ago and know that Google DNS and others are translating correctly. Would you please help us assist your customers?<o:p></o:p></span></p><p><span style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>The correct resolutions are:<br><a href="http://vxn.datawire.net/">vxn.datawire.net</a> 216.220.36.75<br><a href="http://vxn1.datawire.net/">vxn1.datawire.net</a> 205.167.140.10<br><a href="http://vxn2.datawire.net/">vxn2.datawire.net</a> 64.243.142.36<br><a href="http://vxn3.datawire.net/">vxn3.datawire.net</a> 206.112.91.167<br><a href="http://vxn4.datawire.net/">vxn4.datawire.net</a> 63.240.199.76<o:p></o:p></span></p><p><span style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>If you are resolving it as anything starting with 45.x.x.x, it is incorrect. Please feel free to compare to the Google DNS resolution for confirmation.<o:p></o:p></span></p><p><span style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>Please either reply all or call First Data's Network Operations at 888-377-8726 Option 3.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#004165'><snip><o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><br>First Data, <i>240 North Roosevelt Av</i></span><i><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D'> <o:p></o:p></span></i></p><p class=MsoNormal><i><span style='color:#1F497D'>Chandler, Arizona 85226</span></i><span style='color:#1F497D'><br><br></span><o:p></o:p></p><p class=MsoNormal>==================<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>That kind of confirms that the TTL for the 45.x.x.x record(s) were a bit too long – if they had been short, like they are now at 300 seconds, the issue would mostly have cleared up.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Outages <outages-bounces@outages.org> <b>On Behalf Of </b>frnkblk--- via Outages<br><b>Sent:</b> Friday, July 13, 2018 9:56 PM<br><b>To:</b> 'Luke Guillory' <lguillory@reservetele.com>; jayson@peakinter.net; outages@outages.org<br><b>Subject:</b> Re: [outages] Problem with credit card machine processing? "Datawire"<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Yes, we learned of issues late Wednesday morning after receiving reports from two and then three business customers. Indications suggest the issue started Tuesday evening. One local Dairy Queen and another 20 minutes away couldn’t accept credit cards on Wednesday.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The request to preform DNS flushes of vxn.datawire.net came to us Thursday afternoon from two of three customers who (eventually) called their credit card partners/processors. So we flushed our (ISP) caches and then encouraged those customers to power cycle their router and then their credit card machines, but that wasn’t 100% successful for them, either. At that point we directed them back to their credit card partners/processors. It was interesting to see DNS resolution for vxn.datawire.net pointing to a mixture of 216.220.36.75 (vxn.datawire.net) and 45.227.252.17 (hosting-by.net4web.org). Maybe it's normal that they have multiple, but on Wednesday it was just 216.220.36.75. The TTL for 45.227.252.17 was much longer (over 430,000) than 216.220.36.75 (about 300 seconds) and had a bad SSL certificate for <a href="https://vxn.datawire.net">https://vxn.datawire.net</a>. I suspect they moved some operations to another data center, but made a mistake with TTL.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>All told we probably heard from six or seven different businesses.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>More here:<o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/ExecPro/status/1016860164983611392">https://twitter.com/ExecPro/status/1016860164983611392</a><o:p></o:p></p><p class=MsoNormal><a href="https://status.cayan.com/issues/5b45477e8dc35afae9000fe6">https://status.cayan.com/issues/5b45477e8dc35afae9000fe6</a><o:p></o:p></p><p class=MsoNormal><a href="https://status.cayan.com/issues/5b4546508dc35a5975000fdc">https://status.cayan.com/issues/5b4546508dc35a5975000fdc</a><o:p></o:p></p><p class=MsoNormal><a href="https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7">https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7</a><o:p></o:p></p><p class=MsoNormal><a href="https://status.cayan.com/issues/5b478b918dc35aff310030c9">https://status.cayan.com/issues/5b478b918dc35aff310030c9</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/TriphenTech/status/1016852856408690693">https://twitter.com/TriphenTech/status/1016852856408690693</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/C_Forrest/status/1017819893704593410">https://twitter.com/C_Forrest/status/1017819893704593410</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/Vicinity_7/status/1017800989347401728">https://twitter.com/Vicinity_7/status/1017800989347401728</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/pokehbar/status/1017796090052128769">https://twitter.com/pokehbar/status/1017796090052128769</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/glyngh/status/1017790958610493440">https://twitter.com/glyngh/status/1017790958610493440</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/tallbaby21/status/1017121159526133760">https://twitter.com/tallbaby21/status/1017121159526133760</a><o:p></o:p></p><p class=MsoNormal><a href="https://twitter.com/devin_ledude/status/1017451556000522241">https://twitter.com/devin_ledude/status/1017451556000522241</a><o:p></o:p></p><p class=MsoNormal><a href="https://status.cayan.com/issues/5b478ba38dc35a3da80030d9">https://status.cayan.com/issues/5b478ba38dc35a3da80030d9</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText># whob 216.220.36.75<o:p></o:p></p><p class=MsoPlainText>IP: 216.220.36.75<o:p></o:p></p><p class=MsoPlainText>Origin-AS: 12188<o:p></o:p></p><p class=MsoPlainText>Prefix: 216.220.32.0/20<o:p></o:p></p><p class=MsoPlainText>AS-Path: 18106 6939 12188<o:p></o:p></p><p class=MsoPlainText>AS-Org-Name: Q9 Networks Inc.<o:p></o:p></p><p class=MsoPlainText>Org-Name: Q9 Networks Inc.<o:p></o:p></p><p class=MsoPlainText>Net-Name: Q9-NET1<o:p></o:p></p><p class=MsoPlainText>Cache-Date: 1531374425<o:p></o:p></p><p class=MsoPlainText>Latitude: 43.508330<o:p></o:p></p><p class=MsoPlainText>Longitude: -79.883333<o:p></o:p></p><p class=MsoPlainText>City: Milton<o:p></o:p></p><p class=MsoPlainText>Region: Ontario<o:p></o:p></p><p class=MsoPlainText>Country: Canada<o:p></o:p></p><p class=MsoPlainText>Country-Code: CA<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText># whob 45.227.252.17<o:p></o:p></p><p class=MsoPlainText>IP: 45.227.252.17<o:p></o:p></p><p class=MsoPlainText>Origin-AS: 58271<o:p></o:p></p><p class=MsoPlainText>Prefix: 45.227.252.0/24<o:p></o:p></p><p class=MsoPlainText>AS-Path: 34224 12389 44125 201765 48882 58271<o:p></o:p></p><p class=MsoPlainText>AS-Org-Name: VSERVER-AS<o:p></o:p></p><p class=MsoPlainText>Org-Name: This network range is not fully allocated to APNIC.<o:p></o:p></p><p class=MsoPlainText>Net-Name: IANA-NETBLOCK-45<o:p></o:p></p><p class=MsoPlainText>Cache-Date: 1531374425<o:p></o:p></p><p class=MsoPlainText>Latitude: 0.000000<o:p></o:p></p><p class=MsoPlainText>Longitude: 0.000000<o:p></o:p></p><p class=MsoPlainText>City: NULL<o:p></o:p></p><p class=MsoPlainText>Region: NULL<o:p></o:p></p><p class=MsoPlainText>Country: NULL<o:p></o:p></p><p class=MsoPlainText>Country-Code: NULL<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Outages <<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>> <b>On Behalf Of </b>Luke Guillory via Outages<br><b>Sent:</b> Friday, July 13, 2018 9:18 PM<br><b>To:</b> <a href="mailto:jayson@peakinter.net">jayson@peakinter.net</a>; <a href="mailto:outages@outages.org">outages@outages.org</a><br><b>Subject:</b> Re: [outages] Problem with credit card machine processing? "Datawire"<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=imprintuniqueid>We had a customer call saying they needed is to clear dns cache because they couldn’t process CCs.<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'>One of my guys read about the large outage so when it came in we knew it wasn’t anything to do with us. <o:p></o:p></p><div id=AppleMailSignature><p class=MsoNormal><i>Sent from my iPhone</i><o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><br>On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'>Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal. That turned into 2, 3, and a handful more. </span><span style='font-size:9.0pt;font-family:"Arial",sans-serif'><o:p></o:p></span></p></div><div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'>We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ). They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down. <o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'>Anyone else seeing anything like this? Perhaps it could save you chasing your tail as well.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'>Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this? Hmm... <o:p></o:p></span></p><div><div><div><div><div><div><div><div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'> <o:p></o:p></span></p></div></div></div></div></div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'>Jayson<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222'>Peak Internet<o:p></o:p></span></p></div></div></div></div></div></div></div></div></div></blockquote></div><p class=imprintuniqueid> <o:p></o:p></p><table class=MsoNormalTable border=0 cellspacing=0 cellpadding=0 width="100%" style='width:100.0%;border-collapse:collapse'><tr><td width=240 valign=top style='width:2.5in;border:none;border-top:solid #CECECE 1.0pt;padding:7.5pt 0in 0in 0in'><table class=MsoNormalTable border=0 cellspacing=0 cellpadding=0><tr><td style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Verdana",sans-serif;color:#33358B'>Luke</span><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#A0A0A0'> </span><span style='font-size:12.0pt;font-family:"Verdana",sans-serif;color:#33358B'>Guillory</span><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#A0A0A0'><o:p></o:p></span></p></td></tr><tr><td style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#EE5526'>Vice President – Technology and Innovation</span><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#A0A0A0'><o:p></o:p></span></p></td></tr></table><p class=imprintuniqueid><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#A0A0A0'> <o:p></o:p></span></p></td><td valign=top style='border-top:solid #CECECE 1.0pt;border-left:solid #CECECE 1.0pt;border-bottom:none;border-right:none;padding:7.5pt 0in 7.5pt 7.5pt'><p class=MsoNormal><span style='font-size:8.0pt'><a href="http://www.rtconline.com"><span style='text-decoration:none'><img border=0 width=150 height=36 style='width:1.5625in;height:.375in' id="_x0000_i1025" src="cid:image001.jpg@01D41D14.A72813C0"></span></a><o:p></o:p></span></p></td></tr><tr><td style='border:none;border-bottom:solid #CECECE 1.0pt;padding:0in 0in 0in 0in'><table class=MsoNormalTable border=0 cellspacing=0 cellpadding=0><tr><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'>Tel:<o:p></o:p></span></p></td><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'>985.536.1212<o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'>Fax:<o:p></o:p></span></p></td><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'>985.536.0300<o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'>Email:<o:p></o:p></span></p></td><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'><a href="mailto:lguillory@reservetele.com">lguillory@reservetele.com</a><o:p></o:p></span></p></td></tr><tr><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'>Web:<o:p></o:p></span></p></td><td valign=top style='padding:0in 0in 0in 0in'><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'><a href="http://www.rtconline.com">www.rtconline.com</a><o:p></o:p></span></p></td></tr></table></td><td valign=top style='border-top:none;border-left:solid #CECECE 1.0pt;border-bottom:solid #CECECE 1.0pt;border-right:none;padding:0in 0in 7.5pt 7.5pt'><p class=MsoNormal><span style='font-size:7.0pt;font-family:"Verdana",sans-serif;color:#5C5B5B'>Reserve Telecommunications <br>100 RTC Dr<br>Reserve, LA 70084</span><o:p></o:p></p></td></tr></table><p class=imprintuniqueid><span style='font-size:7.0pt'> <o:p></o:p></span></p><p class=imprintuniqueid><span style='font-size:7.0pt;color:#FF6600'> <o:p></o:p></span></p><p class=imprintuniqueid style='margin-bottom:12.0pt'><span style='font-size:7.0pt;color:#FF6600'><br></span><strong><span style='font-size:7.0pt;font-family:"Verdana",sans-serif;color:#CFCFCF;background:white'>Disclaimer:</span></strong><span style='font-size:7.0pt;font-family:"Verdana",sans-serif;color:#CFCFCF;background:white'><br>The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory</span><span style='font-size:7.0pt;font-family:"Verdana",sans-serif;color:#CFCFCF'> immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. <span style='background:white'>Luke Guillory</span> therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. </span><span style='font-size:7.0pt;color:#FF6600'><o:p></o:p></span></p><div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages@outages.org">Outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p></div></blockquote></div></div></body></html>