<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Yes, thank you. Thank all of you. Once things seemed to be starting to converge back into function I had to lay down for a while before I made sure everything
I knew about was working.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">So everyone, check your ROA/RPKI records and make sure they’re right, people are starting to actually authenticate advertisements based on them it seems.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Nimrod Levy [mailto:nimrodl@gmail.com]
<br>
<b>Sent:</b> Friday, February 8, 2019 9:24 PM<br>
<b>To:</b> Eric J Esslinger <eesslinger@fpu-tn.com><br>
<b>Subject:</b> Re: [outages] Reachability issues AS7018 <-> AS63267<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Is this working now? I see the valid ROA at this time and the route in 7018.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Feb 8, 2019 at 9:08 PM Eric J Esslinger via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Thanks everyone for the feedback I'm continuing to work on things. I have fixed the ROA/RPKI object after slapping myself for not adding my own AS number as a valid advertiser when I set that up two years ago.<br>
It's showing up valid at some places, invalid at others, I'm not sure how long it will take to overwrite the invalid information.<br>
<br>
I'm working with Charter but they're head scratching as well. Iris is a relatively small provider and hands off to many others very quickly and if I shut down the Charter side link my iris traffic still routes via charter right through the same telia router
trying to get to AT&T.<br>
<br>
I'll take a bit of time to let things propagate and for Charter to push and then probably take this to Nanog to see if I can find someone AT&T over there. Cause their ARIN contacts sure don't answer phones or emails.<br>
<br>
(I don't even remember why I dropped outages and Nanog, I should be on them...)<br>
<br>
<br>
-----Original Message-----<br>
From: Ryan Kearney [mailto:<a href="mailto:ryan@kearney.io" target="_blank">ryan@kearney.io</a>]<br>
Sent: Friday, February 8, 2019 6:29 PM<br>
To: Eric J Esslinger <<a href="mailto:eesslinger@fpu-tn.com" target="_blank">eesslinger@fpu-tn.com</a>><br>
Cc: <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
Subject: Re: [outages] Reachability issues AS7018 <-> AS63267<br>
<br>
You have ROAs for <a href="http://104.171.208.0/20" target="_blank">104.171.208.0/20</a> allowing ASN 3356, 11758, and 20115 to originate that block. AS AS63267 is not listed. Perhaps AT&T is filtering out your advertisement as a result?<br>
<br>
I would suggest fixing your ROA / RPKI objects for that route.<br>
<br>
On Fri, Feb 8, 2019 at 5:40 PM Eric J Esslinger via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br>
><br>
> Let me preface this with I'm sorry for some very sketchy info here and only a couple of traceroutes, but I've got pneumonia and am working from home atm.<br>
><br>
> Howdy. I'm Eric Esslinger, IS Manager at FPU, owner of AS63267 and our<br>
> associated netblock of <a href="http://104.171.208.0/20" target="_blank">104.171.208.0/20</a> We're unable to receive communication from AS7018 (AT&T Incorporated) through either of our upstream providers (Spectrum Enterprise aka Charter, and Iris networks).<br>
><br>
> Looking glass sites in general show our bgp advertisements to be correct.<br>
><br>
> Traceroutes to any site that requires AT&T that we've identified as a problem show the following (in the Charter case, which is our prefer:<br>
><br>
> 1 6 ms 5 ms 6 ms 104.171.208.1<br>
> 2 6 ms 5 ms 5 ms <a href="http://96-33-106-193.static.kgpt.tn.charter.com" target="_blank">
96-33-106-193.static.kgpt.tn.charter.com</a> [96.33.106.193]<br>
> 3 5 ms 5 ms 5 ms <a href="http://dtr02mnchtn-tge-0-7-0-6.mnch.tn.charter.com" target="_blank">
dtr02mnchtn-tge-0-7-0-6.mnch.tn.charter.com</a> [96.34.71.10]<br>
> 4 11 ms 11 ms 11 ms <a href="http://crr01kgpttn-bue-200.kgpt.tn.charter.com" target="_blank">
crr01kgpttn-bue-200.kgpt.tn.charter.com</a> [96.34.69.168]<br>
> 5 20 ms 17 ms 17 ms <a href="http://crr02spbgsc-bue-401.spbg.sc.charter.com" target="_blank">
crr02spbgsc-bue-401.spbg.sc.charter.com</a> [96.34.69.254]<br>
> 6 22 ms 20 ms 24 ms <a href="http://bbr01spbgsc-bue-4.spbg.sc.charter.com" target="_blank">
bbr01spbgsc-bue-4.spbg.sc.charter.com</a> [96.34.2.50]<br>
> 7 21 ms 20 ms 20 ms <a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fcha-b1-link.telia.net&c=E,1,0n5m0mg2rs8SmGqPXFqTY-y-u3Zc6YTKckX6rs0lsaW_KHFsTtj468hGafX54eTuC6lJxT8hHJ3sD5yuIostnhrEXVXZP9MwopTeHvpDyrK2vID0PA,,&typo=1" target="_blank">
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fcha-b1-link.telia.net&c=E,1,0n5m0mg2rs8SmGqPXFqTY-y-u3Zc6YTKckX6rs0lsaW_KHFsTtj468hGafX54eTuC6lJxT8hHJ3sD5yuIostnhrEXVXZP9MwopTeHvpDyrK2vID0PA,,&typo=1</a> [213.248.98.25]<br>
> 8 24 ms 23 ms 26 ms <a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fatl-b22-link.telia.net&c=E,1,FJaGtSZD6d_s3FQz0M2pDmXEhxwOB5mjEicKqV0sj_yTVAE3rdlrt0VEX-eB6myYyAvKicEZ6ChR5q7HTGwbACf5P9lEyT2Mu6w8ywea_eoKvysNF8us&typo=1" target="_blank">
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fatl-b22-link.telia.net&c=E,1,FJaGtSZD6d_s3FQz0M2pDmXEhxwOB5mjEicKqV0sj_yTVAE3rdlrt0VEX-eB6myYyAvKicEZ6ChR5q7HTGwbACf5P9lEyT2Mu6w8ywea_eoKvysNF8us&typo=1</a> [213.155.137.122]<br>
> 9 * * * Request timed out.<br>
><br>
> From a Charter cable customer in town, the same trace gets farther:<br>
><br>
> 1 <a href="http://75-130-85-45.static.kgpt.tn.charter.com" target="_blank">75-130-85-45.static.kgpt.tn.charter.com</a> (75.130.85.45) 1.832 ms<br>
> 1.174 ms 0.982 ms<br>
> 2 * * *<br>
> 3 <a href="http://acr09mnchtn-gbe-2-28.mnch.tn.charter.com" target="_blank">acr09mnchtn-gbe-2-28.mnch.tn.charter.com</a> (96.34.70.72) 13.900 ms<br>
> 12.677 ms 12.753 ms<br>
> 4 <a href="http://96-34-12-55.static.unas.mo.charter.com" target="_blank">96-34-12-55.static.unas.mo.charter.com</a> (96.34.12.55) 16.337 ms<br>
> 14.036 ms 16.950 ms<br>
> 5 <a href="http://96-34-13-27.static.unas.mo.charter.com" target="_blank">96-34-13-27.static.unas.mo.charter.com</a> (96.34.13.27) 12.622 ms<br>
> 15.170 ms 11.405 ms<br>
> 6 <a href="http://96-34-12-133.static.unas.mo.charter.com" target="_blank">96-34-12-133.static.unas.mo.charter.com</a> (96.34.12.133) 19.547 ms<br>
> <a href="http://96-34-12-143.static.unas.mo.charter.com" target="_blank">96-34-12-143.static.unas.mo.charter.com</a> (96.34.12.143) 23.507 ms<br>
> 19.327 ms<br>
> 7 <a href="http://crr02spbgsc-bue-401.spbg.sc.charter.com" target="_blank">crr02spbgsc-bue-401.spbg.sc.charter.com</a> (96.34.69.254) 23.754 ms<br>
> 31.793 ms 24.025 ms<br>
> 8 <a href="http://bbr01spbgsc-bue-4.spbg.sc.charter.com" target="_blank">bbr01spbgsc-bue-4.spbg.sc.charter.com</a> (96.34.2.50) 30.195 ms<br>
> 23.882 ms 28.211 ms<br>
> 9<br>
> <a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fcha-b1-link.telia" target="_blank">
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fcha-b1-link.telia</a>.<br>
> net&c=E,1,vYtDy2_lF_5tK6YSF5pBvPMuhLPAbFJ94XtFX7ALWzmsg6hnmcb0LCLb8YrM<br>
> uklnW8LT2ngwpjEe6wWaHTDLvWZhW9FJXkow5p5SUiGMqV9ja50,&typo=1<br>
> (213.248.98.25) 39.141 ms 39.260 ms 38.552 ms<br>
> 10<br>
> <a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fatl-b22-link.telia" target="_blank">
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fatl-b22-link.telia</a><br>
> .net&c=E,1,cjSat-D2lK8sGVHUDb42ZtdtBdiuMhVyvm1MwyF0mgKlZxbMb8SiSvA5uSk<br>
> jFhvqsYGoq8HuqHSESpH5tnHp4JZQQKjqIwWMk7TgP6xh_LDbTJJN&typo=1<br>
> (213.155.137.122) 40.886 ms 44.736 ms 39.673 ms<br>
> 11 192.205.33.41 (192.205.33.41) 54.064 ms 45.201 ms 44.614 ms<br>
> (continues)<br>
><br>
> So at a minimum there is a problem at the 192.205.33.41 device.<br>
> This is currently having a major impact on us, as both a provider and the Electric, Water, Sewer, and Gas utility for Fayetteville and Lincoln County TN, the Fayetteville/Lincoln County Tennessee hospital, City and County municipal government, the Police,
Sheriff's department and 911, local EMA, and 4 of the 5 biggest banks, just to cover the more critical people affected.<br>
><br>
> Does AT&T have anyone on this list that can help us out here? We've been beating against a brick wall for two days, hampered somewhat by my having walking pneumonia. Most of our contacts with AT&T are met with the standard brick wall of 'you're not our customer'.<br>
><br>
> If anyone wants a reachability test, feel free to ping 104.171.208.1<br>
><br>
> I can be contacted off list via <a href="mailto:eesslinger@fpu-tn.com" target="_blank">
eesslinger@fpu-tn.com</a><br>
><br>
> Thanks for your attention.<br>
><br>
> Eric Esslinger<br>
> Information Services Manager<br>
> Fayetteville Public Utilities<br>
> 408 College St W<br>
> Fayetteville, TN 37334<br>
> 931-433-1522<br>
><br>
> This message may contain confidential and/or proprietary information and is intended for the person/entity to whom it was originally addressed. Any use by others is strictly prohibited.<br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
> <a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2f" target="_blank">
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2f</a><br>
> mailman%2flistinfo%2foutages&c=E,1,ggkzxy9Dof9rMnhObjpwDGLTyUza1fl-Wq8<br>
> IyNcVj0MThxFD3di_juigoULPiT3L6_ljlE1_BXx2ZXvnpyvGrKYH44iwmJw4lv4clrkor<br>
> sQ,&typo=1<br>
<br>
This message may contain confidential and/or proprietary information and is intended for the person/entity to whom it was originally addressed. Any use by others is strictly prohibited.<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2fmailman%2flistinfo%2foutages&c=E,1,wGXdylKlUq-T8-jBT5S0_Mg319Bch97jpEm_DEIUjZ7OXGr9KHZaFK5zEIZUIOtv9_m8FD7Tlh--K0me6ZmQt50YXJfPjkwR1EqrRoiO97AUDqL3WKDlMJ5qfpU,&typo=1" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="2">This message may contain confidential and/or proprietary information and is intended for the person/entity to whom it was originally addressed. Any use by others is strictly prohibited.<br>
</font>
</body>
</html>