[outages] Problem with credit card machine processing? "Datawire"

Mon Jul 16 15:52:44 EDT 2018

Just received this afternoon:

==================

Support Team,

You have several business customers being affected by an ongoing issue. In order to resolve this, First Data is requesting that you clear the cache on all DNS servers being used to support them. We propagated a correction over 16 hours ago and know that Google DNS and others are translating correctly. Would you please help us assist your customers?

The correct resolutions are:
vxn.datawire.net <http://vxn.datawire.net/>  216.220.36.75
vxn1.datawire.net <http://vxn1.datawire.net/>  205.167.140.10
vxn2.datawire.net <http://vxn2.datawire.net/>  64.243.142.36
vxn3.datawire.net <http://vxn3.datawire.net/>  206.112.91.167
vxn4.datawire.net <http://vxn4.datawire.net/>  63.240.199.76

If you are resolving it as anything starting with 45.x.x.x, it is incorrect. Please feel free to compare to the Google DNS resolution for confirmation.

Please either reply all or call First Data's Network Operations at 888-377-8726 Option 3.

<snip>

First Data, 240 North Roosevelt Av 

Chandler, Arizona 85226

==================

That kind of confirms that the TTL for the 45.x.x.x record(s) were a bit too long – if they had been short, like they are now at 300 seconds, the issue would mostly have cleared up.

From: Outages <outages-bounces at outages.org> On Behalf Of frnkblk--- via Outages
Sent: Friday, July 13, 2018 9:56 PM
To: 'Luke Guillory' <lguillory at reservetele.com>; jayson at peakinter.net; outages at outages.org
Subject: Re: [outages] Problem with credit card machine processing? "Datawire"

Yes, we learned of issues late Wednesday morning after receiving reports from two and then three business customers.  Indications suggest the issue started Tuesday evening.   One local Dairy Queen and another 20 minutes away couldn’t accept credit cards on Wednesday.

The request to preform DNS flushes of vxn.datawire.net came to us Thursday afternoon from two of three customers who (eventually) called their credit card partners/processors.  So we flushed our (ISP) caches and then encouraged those customers to power cycle their router and then their credit card machines, but that wasn’t 100% successful for them, either.  At that point we directed them back to their credit card partners/processors. It was interesting to see DNS resolution for vxn.datawire.net pointing to a mixture of 216.220.36.75 (vxn.datawire.net) and 45.227.252.17 (hosting-by.net4web.org).  Maybe it's normal that they have multiple, but on Wednesday it was just 216.220.36.75. The TTL for 45.227.252.17 was much longer (over 430,000) than 216.220.36.75 (about 300 seconds) and had a bad SSL certificate for https://vxn.datawire.net.  I suspect they moved some operations to another data center, but made a mistake with TTL.

All told we probably heard from six or seven different businesses.

More here:

https://twitter.com/ExecPro/status/1016860164983611392

https://status.cayan.com/issues/5b45477e8dc35afae9000fe6

https://status.cayan.com/issues/5b4546508dc35a5975000fdc

https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7

https://status.cayan.com/issues/5b478b918dc35aff310030c9

https://twitter.com/TriphenTech/status/1016852856408690693

https://twitter.com/C_Forrest/status/1017819893704593410

https://twitter.com/Vicinity_7/status/1017800989347401728

https://twitter.com/pokehbar/status/1017796090052128769

https://twitter.com/glyngh/status/1017790958610493440

https://twitter.com/tallbaby21/status/1017121159526133760

https://twitter.com/devin_ledude/status/1017451556000522241

https://status.cayan.com/issues/5b478ba38dc35a3da80030d9

Frank 

# whob 216.220.36.75

IP: 216.220.36.75

Origin-AS: 12188

Prefix: 216.220.32.0/20

AS-Path: 18106 6939 12188

AS-Org-Name: Q9 Networks Inc.

Org-Name: Q9 Networks Inc.

Net-Name: Q9-NET1

Cache-Date: 1531374425

Latitude: 43.508330

Longitude: -79.883333

City: Milton

Region: Ontario

Country: Canada

Country-Code: CA

# whob 45.227.252.17

IP: 45.227.252.17

Origin-AS: 58271

Prefix: 45.227.252.0/24

AS-Path: 34224 12389 44125 201765 48882 58271

AS-Org-Name: VSERVER-AS

Org-Name: This network range is not fully allocated to APNIC.

Net-Name: IANA-NETBLOCK-45

Cache-Date: 1531374425

Latitude: 0.000000

Longitude: 0.000000

City: NULL

Region: NULL

Country: NULL

Country-Code: NULL

From: Outages <outages-bounces at outages.org <mailto:outages-bounces at outages.org> > On Behalf Of Luke Guillory via Outages
Sent: Friday, July 13, 2018 9:18 PM
To: jayson at peakinter.net <mailto:jayson at peakinter.net> ; outages at outages.org <mailto:outages at outages.org> 
Subject: Re: [outages] Problem with credit card machine processing? "Datawire"

We had a customer call saying they needed is to clear dns cache because they couldn’t process CCs.

One of my guys read about the large outage so when it came in we knew it wasn’t anything to do with us. 

Sent from my iPhone

On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <outages at outages.org <mailto:outages at outages.org> > wrote:

Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal.  That turned into 2, 3, and a handful more.  

We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ).  They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down. 

Anyone else seeing anything like this?  Perhaps it could save you chasing your tail as well.

Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this?  Hmm... 

Jayson

Peak Internet

Luke Guillory

Vice President – Technology and Innovation

 <http://www.rtconline.com> 

Tel:

985.536.1212

Fax:

985.536.0300

Email:

lguillory at reservetele.com <mailto:lguillory at reservetele.com> 

Web:

www.rtconline.com <http://www.rtconline.com> 

Reserve Telecommunications 
100 RTC Dr
Reserve, LA 70084

Disclaimer:
The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. 

_______________________________________________
Outages mailing list
Outages at outages.org <mailto:Outages at outages.org> 
https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20180716/fdece042/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 23559 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20180716/fdece042/attachment.jpg>