[Outages-discussion] [outages] Problem with credit card machine processing? "Datawire"

Sun Aug 5 00:25:23 EDT 2018

Yup. 

There’s nothing like a little passively gathered evidence...
c/o Farsight’s passive DNS service.

;;  bailiwick: datawire.net.
;;      count: 4
;; first seen: 2018-07-10 23:44:11 -0000
;;  last seen: 2018-07-13 00:55:39 -0000
vxn.datawire.net. IN A 45.227.252.17

;;  bailiwick: datawire.net.
;;      count: 5
;; first seen: 2018-07-10 23:44:12 -0000
;;  last seen: 2018-07-13 00:57:51 -0000
vxn1.datawire.net. IN A 45.227.252.17

;;  bailiwick: datawire.net.
;;      count: 2
;; first seen: 2018-07-10 23:44:12 -0000
;;  last seen: 2018-07-10 23:44:12 -0000
vxn2.datawire.net. IN A 45.227.252.17

;;  bailiwick: datawire.net.
;;      count: 6
;; first seen: 2018-07-10 23:44:13 -0000
;;  last seen: 2018-07-13 00:56:07 -0000
vxn3.datawire.net. IN A 45.227.252.17

;;  bailiwick: datawire.net.
;;      count: 9
;; first seen: 2018-07-10 23:44:14 -0000
;;  last seen: 2018-07-13 00:49:06 -0000
vxn4.datawire.net. IN A 45.227.252.17

prod.ssl53.com. IN A 45.227.252.17
vxn.datawire.net. IN A 45.227.252.17
vxn1.datawire.net. IN A 45.227.252.17
vxn2.datawire.net. IN A 45.227.252.17
vxn3.datawire.net. IN A 45.227.252.17
vxn4.datawire.net. IN A 45.227.252.17
;;; Returned 6 RRs in 0.02 seconds.
;;; DNSDB

Data wire is First Data, however, that outlier above.... 

;;  bailiwick: ssl53.com.
;;      count: 17
;; first seen: 2018-07-13 00:48:09 -0000
;;  last seen: 2018-07-13 01:00:26 -0000
prod.ssl53.com. IN A 45.227.252.17

= Vantiv, LLC, isn’t that Worldpay?

Hmmm. 

> On 5 Aug 2018, at 11:56, <frnkblk at iname.com> <frnkblk at iname.com> wrote:
> 
> Looks like Datawire did sweep it under the rug – here’s a Dyn blog written by Doug Madory about how the IP address space for Datawire’s nameservers were hijacked for a short time:
> https://dyn.com/blog/bgp-dns-hijacks-target-payment-systems/
> The July 10 incident would be Tuesday afternoon/early evening in the U.S.
>  
> Now its’ very clear why the payment processors wanted ISPs to flush Datawire’s host entries.
>  
> Frank
>  
> From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf Of Frank Bulk
> Sent: Tuesday, July 17, 2018 3:41 PM
> To: outages-discussion at outages.org
> Subject: Re: [Outages-discussion] [outages] Problem with credit card machine processing? "Datawire"
>  
> I had assumed that the VPS provider was their DR solution. =)
>  
> Frank
>  
> From: Randy McAnally <rsm at fast-serv.com> 
> Sent: Tuesday, July 17, 2018 3:07 PM
> To: Frank Bulk <frnkblk at iname.com>
> Cc: outages-discussion at outages.org
> Subject: Re: [outages] Problem with credit card machine processing? "Datawire"
>  
> 45.227.252.17 + high TTL + ukraine VPS provider
> 
> did first data just sweep this under the rug?
> 
>  
> 
> On 07/16/2018 12:52 pm, Frank Bulk via Outages wrote:
> 
> Just received this afternoon:
>  
> ==================
> Support Team,
> 
> You have several business customers being affected by an ongoing issue. In order to resolve this, First Data is requesting that you clear the cache on all DNS servers being used to support them. We propagated a correction over 16 hours ago and know that Google DNS and others are translating correctly. Would you please help us assist your customers?
> 
> The correct resolutions are:
> vxn.datawire.net 216.220.36.75
> vxn1.datawire.net 205.167.140.10
> vxn2.datawire.net 64.243.142.36
> vxn3.datawire.net 206.112.91.167
> vxn4.datawire.net 63.240.199.76
> 
> If you are resolving it as anything starting with 45.x.x.x, it is incorrect. Please feel free to compare to the Google DNS resolution for confirmation.
> 
> Please either reply all or call First Data's Network Operations at 888-377-8726 Option 3.
> 
> <snip>
> 
> First Data, 240 North Roosevelt Av 
> Chandler, Arizona 85226
> 
> 
> ==================
>  
> That kind of confirms that the TTL for the 45.x.x.x record(s) were a bit too long – if they had been short, like they are now at 300 seconds, the issue would mostly have cleared up.
>  
> From: Outages <outages-bounces at outages.org> On Behalf Of frnkblk--- via Outages
> Sent: Friday, July 13, 2018 9:56 PM
> To: 'Luke Guillory' <lguillory at reservetele.com>; jayson at peakinter.net; outages at outages.org
> Subject: Re: [outages] Problem with credit card machine processing? "Datawire"
>  
> Yes, we learned of issues late Wednesday morning after receiving reports from two and then three business customers.  Indications suggest the issue started Tuesday evening.   One local Dairy Queen and another 20 minutes away couldn't accept credit cards on Wednesday.
>  
> The request to preform DNS flushes of vxn.datawire.net came to us Thursday afternoon from two of three customers who (eventually) called their credit card partners/processors.  So we flushed our (ISP) caches and then encouraged those customers to power cycle their router and then their credit card machines, but that wasn't 100% successful for them, either.  At that point we directed them back to their credit card partners/processors. It was interesting to see DNS resolution for vxn.datawire.net pointing to a mixture of 216.220.36.75 (vxn.datawire.net) and 45.227.252.17 (hosting-by.net4web.org).  Maybe it's normal that they have multiple, but on Wednesday it was just 216.220.36.75. The TTL for 45.227.252.17 was much longer (over 430,000) than 216.220.36.75 (about 300 seconds) and had a bad SSL certificate for https://vxn.datawire.net.  I suspect they moved some operations to another data center, but made a mistake with TTL.
>  
> All told we probably heard from six or seven different businesses.
>  
> More here:
> https://twitter.com/ExecPro/status/1016860164983611392
> https://status.cayan.com/issues/5b45477e8dc35afae9000fe6
> https://status.cayan.com/issues/5b4546508dc35a5975000fdc
> https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7
> https://status.cayan.com/issues/5b478b918dc35aff310030c9
> https://twitter.com/TriphenTech/status/1016852856408690693
> https://twitter.com/C_Forrest/status/1017819893704593410
> https://twitter.com/Vicinity_7/status/1017800989347401728
> https://twitter.com/pokehbar/status/1017796090052128769
> https://twitter.com/glyngh/status/1017790958610493440
> https://twitter.com/tallbaby21/status/1017121159526133760
> https://twitter.com/devin_ledude/status/1017451556000522241
> https://status.cayan.com/issues/5b478ba38dc35a3da80030d9
>  
> Frank
>  
>  
> 
> # whob 216.220.36.75
> 
> IP: 216.220.36.75
> 
> Origin-AS: 12188
> 
> Prefix: 216.220.32.0/20
> 
> AS-Path: 18106 6939 12188
> 
> AS-Org-Name: Q9 Networks Inc.
> 
> Org-Name: Q9 Networks Inc.
> 
> Net-Name: Q9-NET1
> 
> Cache-Date: 1531374425
> 
> Latitude: 43.508330
> 
> Longitude: -79.883333
> 
> City: Milton
> 
> Region: Ontario
> 
> Country: Canada
> 
> Country-Code: CA
> 
>  
> 
> # whob 45.227.252.17
> 
> IP: 45.227.252.17
> 
> Origin-AS: 58271
> 
> Prefix: 45.227.252.0/24
> 
> AS-Path: 34224 12389 44125 201765 48882 58271
> 
> AS-Org-Name: VSERVER-AS
> 
> Org-Name: This network range is not fully allocated to APNIC.
> 
> Net-Name: IANA-NETBLOCK-45
> 
> Cache-Date: 1531374425
> 
> Latitude: 0.000000
> 
> Longitude: 0.000000
> 
> City: NULL
> 
> Region: NULL
> 
> Country: NULL
> 
> Country-Code: NULL
> 
>  
> From: Outages <outages-bounces at outages.org> On Behalf Of Luke Guillory via Outages
> Sent: Friday, July 13, 2018 9:18 PM
> To: jayson at peakinter.net; outages at outages.org
> Subject: Re: [outages] Problem with credit card machine processing? "Datawire"
>  
> We had a customer call saying they needed is to clear dns cache because they couldn't process CCs.
> 
>  
> One of my guys read about the large outage so when it came in we knew it wasn't anything to do with us. 
> 
> Sent from my iPhone
> 
> On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <outages at outages.org> wrote:
> 
> Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal.  That turned into 2, 3, and a handful more.  
>  
> We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ).  They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down.
>  
> Anyone else seeing anything like this?  Perhaps it could save you chasing your tail as well.
>  
> Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this?  Hmm...
>  
>  
> Jayson
> Peak Internet
>  
> 
> Luke Guillory
> Vice President – Technology and Innovation
>  
> 
> 
> Tel:
> 985.536.1212
> Fax:
> 985.536.0300
> Email:
> lguillory at reservetele.com
> Web:
> www.rtconline.com
> Reserve Telecommunications 
> 100 RTC Dr
> Reserve, LA 70084
>  
> 
>  
> 
> 
> Disclaimer:
> The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>  
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20180805/6a765cf5/attachment-0001.html>