[outages] Cybersource Payment Gateway

[Mike Castoro]

Post mortem from Cybersource (and likely other payment gateways) on what happened on Tuesday...

"This was due to a network routing issue introduced by a third-party regional telecommunications service provider."



*Incident Title*
Transaction Failures via Multiple CyberSource Services

*Status*
Resolved

*Incident Date/Duration*
Start:     2018/07/17 16:48 (23:48 GMT)
End:       2018/07/17 18:52 (01:52 GMT)
Duration:  2 hours, 4 minutes.

*Summary*
CyberSource merchants experienced transaction failures utilizing multiple CyberSource service offerings dependent upon external gateways/processors. This was due to a network routing issue introduced by a third-party regional telecommunications service provider. CyberSource teams attempted to utilize the alternate datacenter and redundant network routes; however, this action failed to mitigate the issue.

*Impact*
During the incident period, merchants processing transactions via CyberSource datacenters received ESYSTEM and/or 150 error messages. Any transaction(s) that received these error messages failed processing.

*Root Cause*
A regional third-party telecommunications service provider introduced routing changes that affected communications between CyberSource and third-party gateways/processors. As this new routing information was more specific than the routing information advertised by       CyberSource, reply information for requests initiated by CyberSource were not routed back as expected. As a result, CyberSource was unable to process these requests that were dependent on external processors/gateways.

*Related and Contributing Factors*
CyberSource monitoring systems immediately identified the issue at the application layer. As the impact emanated from an uncommunicated change instituted by the third-party telecommunications service, an increased amount of time was needed for determination of root cause and subsequent mitigation actions.
This situation (lost return traffic) did not trigger alarms on the network components as they were functioning as designed. As no alerts are in place to monitor improper routing, there were no observed error conditions in any network device to pinpoint the issue for prompt remediation.

*Resolution*
CyberSource mitigated the incident by installing filters on CyberSource datacenter routers to ignore the third-party telecommunications route announcements. Once CyberSource installed the filters, expected transaction routing and processing resumed.



*Action Taken for Remediation/Improvement*
The following action(s) by CyberSource were taken and/or are planned to quickly identify and prevent an incident recurrence-
CyberSource has implemented a change to ensure this particular third-party network is not able to cause this kind of issue in the future.
CyberSource to include additional filtering on all existing and future traffic routes to prevent the inadvertent introduction of incorrect routing information.
CyberSource is working with the third-party telecommunications network responsible for this impact to understand reasons behind this change, and identify process improvements to prevent a recurrence.

***Timeline and Actions Taken***
List of symptoms encountered and major actions taken by CyberSource during the incident.
All times reported in Pacific Time (PST/PDT) and Greenwich Mean Time (GMT) in parenthesis.

* Jun 17 16:48 (23:48) - Issue seen on outbound processor connection via CyberSource datacenters. Teams observe an application issue with respect to outbound traffic communication.
Note: No network impact observed at this time.

* 16:48 - 16:52
(23:48 - 23:52) - CyberSource fails transaction processing to alternate datacenter. Issue continues.

* 17:06 (00:06) - CyberSource opens bridge for additional troubleshooting.

* 17:49 (00:49) - CyberSource incorporates a routing filter to a single datacenter router. CyberSource observed stability processing through that device.

* 18:52 (01:52)- Additional filters are installed on all edge devices. Errors on all processing platforms cease.

Mike Castoro | Director of IT
Adrianna Papell Group | 646-723-3885

From: Mike Castoro
Sent: Tuesday, July 17, 2018 10:24 PM
To: Phillip Lewis <phillipl at joinvip.com>
Cc: outages at outages.org
Subject: Re: [outages] Cybersource Payment Gateway

Thanks Phillip. Processing is back to normal for me as of 9:36pm ET.
Mike Castoro
Director of IT
Adrianna Papell
(646) 723-3885
mcastoro at adriannapapell.com<mailto:mcastoro at adriannapapell.com>

________________________________
From: Phillip Lewis <phillipl at joinvip.com<mailto:phillipl at joinvip.com>>
Sent: Tuesday, July 17, 2018 9:05:41 PM
To: Mike Castoro
Cc: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] Cybersource Payment Gateway

Confirmed.
I got through to support. Major Processor outage affecting the card processors (not Cybersource directly). First Data and others involved.



On Tue, Jul 17, 2018 at 9:02 PM Mike Castoro via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
Anyone use Cybersource (a VISA Company) for credit card payment gateway and having issues?  Been getting failed authorizations since 7:55PM EST. Been on hold with Support for a while.  No alerts or notifications sent by Cybersource yet.  Even tried to enter authorize directly in the virtual terminal.

Reason Code 150 - General system failure
The payment processor returned an unexpected result: Connection reset: gpn

Apologies if this isn't the place for this type of issues (I saw a similar issue last week).

Mike Castoro | Director of IT
Adrianna Papell Group | New York, NY 10018
(646) 723-3885 | mcastoro at adriannapapell.com<mailto:mcastoro at adriannapapell.com>
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages


--
Phillip Lewis
Director of Operations
VIP Communications Inc
Office | (+1) 703-708-1515 Ext 3028
Email | phillipl at joinvip.com<mailto:phillipl at joinvip.com>
Website | www.joinvip.com<http://www.joinvip.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20180720/b26ac012/attachment.htm>