<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Century Gothic";
panose-1:2 11 5 2 2 2 2 2 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"Arial Black";
panose-1:2 11 10 4 2 1 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
h1
{mso-style-priority:1;
mso-style-link:"Heading 1 Char";
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:5.7pt;
margin-bottom:.0001pt;
font-size:15.5pt;
font-family:"Segoe UI",sans-serif;}
p.MsoBodyText, li.MsoBodyText, div.MsoBodyText
{mso-style-priority:1;
mso-style-link:"Body Text Char";
margin-top:.4pt;
margin-right:0in;
margin-bottom:0in;
margin-left:5.7pt;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Segoe UI",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:1;
mso-style-link:"Heading 1";
font-family:"Segoe UI",sans-serif;
font-weight:bold;}
span.BodyTextChar
{mso-style-name:"Body Text Char";
mso-style-priority:1;
mso-style-link:"Body Text";
font-family:"Segoe UI",sans-serif;}
p.TableParagraph, li.TableParagraph, div.TableParagraph
{mso-style-name:"Table Paragraph";
mso-style-priority:1;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:47.0pt 86.0pt 64.0pt 80.0pt;}
div.WordSection1
{page:WordSection1;}
@page WordSection2
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection2
{page:WordSection2;}
/* List Definitions */
@list l0
{mso-list-id:1975793436;
mso-list-type:hybrid;
mso-list-template-ids:-474056788 -1475199294 425772836 1005250126 -1111033808 -238093082 -1353175312 815544204 1336042362 -1766139242;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:19.4pt;
text-indent:-13.7pt;
mso-ansi-font-size:10.0pt;
mso-bidi-font-size:10.0pt;
font-family:Symbol;
mso-fareast-font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:62.05pt;
text-indent:-13.7pt;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:104.7pt;
text-indent:-13.7pt;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.35pt;
text-indent:-13.7pt;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:190.0pt;
text-indent:-13.7pt;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:232.7pt;
text-indent:-13.7pt;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:275.35pt;
text-indent:-13.7pt;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:318.0pt;
text-indent:-13.7pt;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\2022;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:360.65pt;
text-indent:-13.7pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1030" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Post mortem from Cybersource (and likely other payment gateways) on what happened on Tuesday…
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">“This was due to a network routing issue introduced by a third-party regional telecommunications service provider.”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:dotted windowtext 3.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Incident Title</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Transaction Failures via Multiple CyberSource Services<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Status</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Resolved<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Incident Date/Duration</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Start: 2018/07/17 16:48 (23:48 GMT)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">End: 2018/07/17 18:52 (01:52 GMT)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Duration: 2 hours, 4 minutes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Summary</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">CyberSource merchants experienced transaction failures utilizing multiple CyberSource service offerings dependent upon external gateways/processors. This was due to a network routing issue introduced by a third-party
regional telecommunications service provider. CyberSource teams attempted to utilize the alternate datacenter and redundant network routes; however, this action failed to mitigate the issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Impact</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">During the incident period, merchants processing transactions via CyberSource datacenters received ESYSTEM and/or 150 error messages. Any transaction(s) that received these error messages failed processing.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Root Cause</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">A regional third-party telecommunications service provider introduced routing changes that affected communications between CyberSource and third-party gateways/processors. As this new routing information was
more specific than the routing information advertised by CyberSource, reply information for requests initiated by CyberSource were not routed back as expected. As a result, CyberSource was unable to process these requests that were dependent on external
processors/gateways.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Related and Contributing Factors</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">CyberSource monitoring systems immediately identified the issue at the application layer. As the impact emanated from an uncommunicated change instituted by the third-party telecommunications service, an increased
amount of time was needed for determination of root cause and subsequent mitigation actions.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">This situation (lost return traffic) did not trigger alarms on the network components as they were functioning as designed. As no alerts are in place to monitor improper routing, there were no observed error
conditions in any network device to pinpoint the issue for prompt remediation.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Resolution</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">CyberSource mitigated the incident by installing filters on CyberSource datacenter routers to ignore the third-party telecommunications route announcements. Once CyberSource installed the filters, expected transaction
routing and processing resumed.<o:p></o:p></span></p>
</div>
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><br clear="all" style="page-break-before:always">
</span>
<div class="WordSection2">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">*<b>Action Taken for Remediation/Improvement</b>*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The following action(s) by CyberSource were taken and/or are planned to quickly identify and prevent an incident recurrence–<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">CyberSource has implemented a change to ensure this particular third-party network is not able to cause this kind of issue in the future.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">CyberSource to include additional filtering on all existing and future traffic routes to prevent the inadvertent introduction of incorrect routing information.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">CyberSource is working with the third-party telecommunications network responsible for this impact to understand reasons behind this change, and identify process improvements to prevent a recurrence.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">***Timeline and Actions Taken***<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">List of symptoms encountered and major actions taken by CyberSource during the incident.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">All times reported in Pacific Time (PST/PDT) and Greenwich Mean Time (GMT) in parenthesis.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">* Jun 17 16:48 (23:48) - Issue seen on outbound processor connection via CyberSource datacenters. Teams observe an application issue with respect to outbound traffic communication.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Note: No network impact observed at this time.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">* 16:48 – 16:52<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">(23:48 – 23:52) - CyberSource fails transaction processing to alternate datacenter. Issue continues.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">* 17:06 (00:06) - CyberSource opens bridge for additional troubleshooting.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">* 17:49 (00:49) - CyberSource incorporates a routing filter to a single datacenter router. CyberSource observed stability processing through that device.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">* 18:52 (01:52)- Additional filters are installed on all edge devices. Errors on all processing platforms cease.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#1F497D">Mike Castoro | Director of IT<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Adrianna Papell Group | 646-723-3885<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Mike Castoro <br>
<b>Sent:</b> Tuesday, July 17, 2018 10:24 PM<br>
<b>To:</b> Phillip Lewis <phillipl@joinvip.com><br>
<b>Cc:</b> outages@outages.org<br>
<b>Subject:</b> Re: [outages] Cybersource Payment Gateway<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Arial",sans-serif;color:black">Thanks Phillip. Processing is back to normal for me as of 9:36pm ET.
<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Mike Castoro<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Director of IT<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Adrianna Papell<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">(646) 723-3885<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><a href="mailto:mcastoro@adriannapapell.com">mcastoro@adriannapapell.com</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Phillip Lewis <<a href="mailto:phillipl@joinvip.com">phillipl@joinvip.com</a>><br>
<b>Sent:</b> Tuesday, July 17, 2018 9:05:41 PM<br>
<b>To:</b> Mike Castoro<br>
<b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Cybersource Payment Gateway</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Confirmed. <o:p></o:p></p>
<div>
<p class="MsoNormal">I got through to support. Major Processor outage affecting the card processors (not Cybersource directly). First Data and others involved.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jul 17, 2018 at 9:02 PM Mike Castoro via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Anyone use Cybersource (a VISA Company) for credit card payment gateway and having issues? Been getting failed authorizations since 7:55PM EST. Been on hold with Support for a
while. No alerts or notifications sent by Cybersource yet. Even tried to enter authorize directly in the virtual terminal.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Reason Code 150 - General system failure<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The payment processor returned an unexpected result: Connection reset: gpn<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Apologies if this isn’t the place for this type of issues (I saw a similar issue last week).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Century Gothic",sans-serif">Mike Castoro</span></b><span style="font-family:"Century Gothic",sans-serif">
</span><span style="font-size:10.0pt;font-family:"Century Gothic",sans-serif">| Director of IT</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Century Gothic",sans-serif">Adrianna Papell Group | New York, NY 10018</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Century Gothic",sans-serif">(646) 723-3885 |
<a href="mailto:mcastoro@adriannapapell.com" target="_blank">mcastoro@adriannapapell.com</a></span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial Black",sans-serif;color:#1F497D;background:white">Phillip Lewis</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#888888"><br>
<span style="background:white">Director of Operations</span><br>
</span><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D;background:white">VIP Communications Inc</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#888888"><br>
<span style="background:white">Office | (</span></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#3333FF;background:white">+1)</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#888888;background:white"> </span>703-708-1515
Ext 3028<span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#888888"><br>
<span style="background:white">Email | </span></span><a href="mailto:phillipl@joinvip.com" target="_blank"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1155CC">phillipl@joinvip.com</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#888888;background:white"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#888888">Website | <a href="http://www.joinvip.com/" target="_blank">www.joinvip.com</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>