[Outages-discussion] [outages] Fwd: Undeliverable: Re: .org whois outage?

Mon Sep 12 03:01:30 EDT 2016

(Moved to outages-discussion, CC'ing Jared -- request for you is at the
end of the email)

I disagree with the assertion that mailman on outages.org is
misconfigured.

The outages.org subscriber reamea.chey at azcom.net.kh appears to be
indirectly triggering this problem.  **I** did not see it, so it may
have been a transient issue, but hard to say.  So let's review the SMTP
conversation history since bounces are nice enough to include it.
Working from the bottom up:

> Received: from puck.nether.net (puck.nether.net [204.42.254.5])       by
>  mx04.online.com.kh (Postfix) with ESMTP id 3BD2537F2B        for
>  <reamea.chey at azcom.net.kh>; Sun, 11 Sep 2016 01:23:55 +0700 (ICT)

puck.nether.net (where outages.org is hosted) contacts
mx04.online.com.kh (an MX record for azcom.net.kh).

> Received: from mx04.online.com.kh (mx04.online.com.kh [203.189.128.14])       by
>  mx04.online.com.kh (Postfix) with SMTP id BF29C1E7862        for
>  <reamea.chey at online.com.kh>; Sun, 11 Sep 2016 01:23:55 +0700 (ICT)

That server then re-writes the delivery address into
reamea.chey at online.com.kh (no idea why) before punting the mail to
something called "red" (a Microsoft Exchange server from the look of
it), which may in fact be the machine itself but with a private address
interface (192.168.1.170) or possibly transparent SMTP forwarding of
some kind:

> Received: from red (192.168.1.170) by red.cogetel.com.kh (192.168.1.170) with
>  Microsoft SMTP Server id 14.3.181.6; Sun, 11 Sep 2016 01:21:02 +0700

The same server then attempts to punt the mail to 192.168.1.172 (another
machine claiming to be the same thing, "red"):

> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Sun, 11 Sep
>  2016 01:21:02 +0700

This infinitely continues until the "hop count" (think traceroute but
with SMTP history), indicating a kind of SMTP redirection loop (i.e.
192.168.1.172 is rejecting what 192.168.1.170 is trying to do, but it
keeps trying because that's how it's configured).  What's interesting
is that the redelivery attempts for this go on for over a day.

So why did _you_ get a copy of this?

When you replied to my Email, you (appropriately) sent a copy to me via
the To: line (which I got), and also CC'd outages at outages.org (which
then sends a copy out each subscriber).  When certain kinds of bounces
happen, they end up going back to the original person who sent the mail.
They're SUPPOSED to go to the MAIL FROM address (which is
outages-bounces at outages.org I assure you (I checked)), but some software
has been known to key off of the From: line instead (this is
particularly common when seeing "magic things" that appear in the SMTP
path that don't make clear indication of what's going on in Received:
headers, or because there is a kind of forwarding happening where the
original MAIL FROM is lost).  Some details are on Wikipedia:

https://en.wikipedia.org/wiki/Bounce_address

mxtoolbox.com also has a "header analyser" tool that can parse Received:
lines and make them a bit more clear.  Here's a link to the tool, and
the analysis in question:

http://mxtoolbox.com/EmailHeaders.aspx
http://mxtoolbox.com/Public/Tools/EmailHeaders.aspx?huid=cea3b136-d1a3-43ea-af3a-439a5710aaaa

Jared -- maybe you could remove reamea.chey at azcom.net.kh for the time
being, or send them an Email directly telling them of the issue (to
forward to whatever mail services provider they use)?

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Sun, Sep 11, 2016 at 11:10:31PM -0700, Aaron C. de Bruyn via Outages wrote:
> Mailman must be set up wrong for outages.  I'm getting bounce messages from
> a handful of users like this one.
> 
> -A
> 
> 
> ---------- Forwarded message ----------
> From: <postmaster at cogetel.com.kh>
> Date: Sun, Sep 11, 2016 at 11:05 PM
> Subject: Undeliverable: Re: [outages] .org whois outage?
> To: aaron at heyaaron.com
> 
> 
> *Delivery has failed to these recipients or groups:*
> 
> reamea.chey at cogetel.com.kh
> A problem occurred during the delivery of this message. Please try to
> resend the message later. If the problem continues, contact your helpdesk.
> 
> The following organization rejected your message: red.cogetel.com.kh.
> 
> 
> 
> 
> 
> 
> *Diagnostic information for administrators:*
> 
> Generating server: cogetel.com.kh
> 
> reamea.chey at cogetel.com.kh
> red.cogetel.com.kh #554 5.4.6 Hop count exceeded - possible mail loop ##
> 
> Original message headers:
> 
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 12:46:48 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 12:28:15 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 12:09:37 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 11:50:14 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 11:26:53 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 11:07:04 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 10:41:20 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 10:08:25 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 12 Sep
>  2016 09:49:43 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Sun, 11 Sep
>  2016 01:21:03 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Sun, 11 Sep
>  2016 01:21:03 +0700
> Received: from RED.cogetel.com.kh (192.168.1.170) by red.cogetel.com.kh
>  (192.168.1.172) with Microsoft SMTP Server (TLS) id 14.3.181.6; Sun, 11 Sep
>  2016 01:21:02 +0700
> Received: from red (192.168.1.170) by red.cogetel.com.kh (192.168.1.170) with
>  Microsoft SMTP Server id 14.3.181.6; Sun, 11 Sep 2016 01:21:02 +0700
> Return-Path: <outages-bounces at outages.org>
> X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mx04.online.com.kh
> X-Spam-Level:
> X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_50,HTML_MESSAGE,
> 	NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_MED,SPF_HELO_PASS,T_DKIM_INVALID,WEIRD_PORT
> 	autolearn=no version=3.3.1
> X-Original-To: reamea.chey at online.com.kh
> Delivered-To: reamea.chey at online.com.kh
> Received: from mx04.online.com.kh (mx04.online.com.kh [203.189.128.14])	by
>  mx04.online.com.kh (Postfix) with SMTP id BF29C1E7862	for
>  <reamea.chey at online.com.kh>; Sun, 11 Sep 2016 01:23:55 +0700 (ICT)
> Received: from puck.nether.net (puck.nether.net [204.42.254.5])	by
>  mx04.online.com.kh (Postfix) with ESMTP id 3BD2537F2B	for
>  <reamea.chey at azcom.net.kh>; Sun, 11 Sep 2016 01:23:55 +0700 (ICT)
> Received: from puck.nether.net (localhost [IPv6:::1])	by puck.nether.net
>  (Postfix) with ESMTP id 27979540990;	Sat, 10 Sep 2016 14:23:47 -0400 (EDT)
> X-Original-To: outages at outages.org
> Delivered-To: outages at outages.org
> Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com
>  [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher
>  ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested)
>  by puck.nether.net (Postfix) with ESMTPS id E143F54097A for
>  <outages at outages.org>; Sat, 10 Sep 2016 14:22:36 -0400 (EDT)
> Received: by mail-oi0-x22b.google.com with SMTP id d191so235490oih.2 for
>  <outages at outages.org>; Sat, 10 Sep 2016 11:22:36 -0700 (PDT)
> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heyaaron.com; s=google;
>  h=mime-version:in-reply-to:references:from:date:message-id:subject:to
>  :cc; bh=RavI9Zh4GzMRBs/5a2hYm2MtmRpmAWzmsEMt+X5Po10=;
>  b=FHgs3sElXNJ8sEQcgcslTwc0bid3W3GPPVzOuRyAJYs4JPW/0DoCks8rM+YL3bTr1i
>  5mI7zN3ZV/ufMAq29fNRd5sNVXiDROL8Xj7MWTHb+U6EHOLKA9UtLUe40iekX8YQLx6f
>  QSOo0G4UWXhzdWSUvmGDvaD47yJkg//SSd2OE=
> X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
>  d=1e100.net; s=20130820;
>  h=x-gm-message-state:mime-version:in-reply-to:references:from:date
>  :message-id:subject:to:cc;
>  bh=RavI9Zh4GzMRBs/5a2hYm2MtmRpmAWzmsEMt+X5Po10=;
>  b=msucx6rRy6uQvrUOR/7Vo+ISSKJQvqapWAQKJ7VzKiW4ZfvrwQ26zqhFLzY3UGv563
>  CW6FeNAGAOI0e07osad/3J2rIbMJfxmQZpPQ8w3/epW723JQQQqOzypPRyTAA1hh0M8q
>  dsQqsNvvmBe4dN8MEAzD3te/4nX30UgZiSCb6/FKUUgqAcjeGVMV8vgZ9zvhc2vAYXNv
>  FadEFYMavaiySaAR2SGbBGw2n0SoCov52XGP0yU5HBQ3J7gyulmG0968Eq+2aTD6kY/A
>  nhsAVUtjRUi+lYCV2X95vClPJeq4pEQgl5AhHg27KFf9pqEz7UGnXSgWDVMeYQgHkM9W
>  BuYA==
> X-Gm-Message-State:
> AE9vXwOcJC/4u2M0bC/14aBvWsdCZvvJUUqZdovO1ZoFkGibHdg06F+ca0c7I0bO/H4PjJc80T3G5AmsVPFu435Y
> X-Received: by 10.157.11.104 with SMTP id p37mr11750000otd.132.1473531756158;
>  Sat, 10 Sep 2016 11:22:36 -0700 (PDT)
> MIME-Version: 1.0
> Received: by 10.107.13.69 with HTTP; Sat, 10 Sep 2016 11:22:20 -0700 (PDT)
> In-Reply-To: <20160910182138.GA28041 at icarus.home.lan>
> References: <CAEE+rGrdSrBgD-pitNaaiH90_uEPFQ1B3fp2s58iGTjRgWNHXQ at mail.gmail.com>
>  <20160910182138.GA28041 at icarus.home.lan>
> Date: Sat, 10 Sep 2016 11:22:20 -0700
> Message-ID: <CAEE+rGo=Z1dBVU-3T3+Y2wSJStN62kdcnEZf5LBPvjUaz_Re4g at mail.gmail.com>
> To: Jeremy Chadwick <jdc at koitsu.org>
> Subject: Re: [outages] .org whois outage?
> X-BeenThere: outages at outages.org
> X-Mailman-Version: 2.1.22
> Precedence: list
> List-Id: "Outages \(planned & unplanned\) Reporting." <outages.outages.org>
> List-Unsubscribe: <https://puck.nether.net/mailman/options/outages>,
>  <mailto:outages-request at outages.org?subject=unsubscribe>
> List-Archive: <https://puck.nether.net/pipermail/outages/>
> List-Post: <mailto:outages at outages.org>
> List-Help: <mailto:outages-request at outages.org?subject=help>
> List-Subscribe: <https://puck.nether.net/mailman/listinfo/outages>,
>  <mailto:outages-request at outages.org?subject=subscribe>
> From: "Aaron C. de Bruyn via Outages" <outages at outages.org>
> Reply-To: "Aaron C. de Bruyn" <aaron at heyaaron.com>
> CC: <outages at outages.org>
> Content-Type: multipart/mixed;
> 	boundary="===============5405249895932320106=="
> Errors-To: outages-bounces at outages.org
> Sender: Outages <outages-bounces at outages.org>
> X-AntiVirus: checked by Vexira MailArmor
> 
> 
> Final-Recipient: rfc822;reamea.chey at cogetel.com.kh
> Action: failed
> Status: 5.4.6
> Diagnostic-Code: smtp;554 5.4.6 Hop count exceeded - possible mail loop
> Remote-MTA: dns;red.cogetel.com.kh
> 
> 
> 
> ---------- Forwarded message ----------
> From: "Aaron C. de Bruyn via Outages" <outages at outages.org>
> To: Jeremy Chadwick <jdc at koitsu.org>
> Cc: <outages at outages.org>
> Date: Sat, 10 Sep 2016 11:22:20 -0700
> Subject: Re: [outages] .org whois outage?
> Appears to be back online now.
> 
> -A
> 
> On Sat, Sep 10, 2016 at 11:21 AM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> 
> > Can confirm.  On FreeBSD using native base system whois:
> >
> > $ truss -f whois koitsu.org
> > ...
> > 28050: socket(PF_INET,SOCK_DGRAM,17)             = 3 (0x3)
> > 28050: connect(3,{ AF_INET 199.15.84.131:1 },16) = 0 (0x0)
> > 28050: getsockname(3,{ AF_INET 192.168.1.51:32474 },0x7fffffffe61c) = 0
> > (0x0)
> > 28050: close(3)                                  = 0 (0x0)
> > 28050: socket(PF_INET,SOCK_STREAM,6)             = 3 (0x3)
> > ^C28050: connect(3,{ AF_INET 199.15.84.131:43 },16) ERR#4 'Interrupted
> > system call'
> > 28050: SIGNAL 2 (SIGINT)
> > 28050: process exit, rval = 0
> >
> > It appears 199.15.84.131 isn't responding on TCP port 43 (WHOIS service
> > port):
> >
> > 11:17:48.119258 IP 192.168.1.51.56576 > 199.15.84.131.43: Flags [S], seq
> > 4226693514, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val
> > 2452140445 ecr 0], length 0
> > 11:17:51.118877 IP 192.168.1.51.56576 > 199.15.84.131.43: Flags [S], seq
> > 4226693514, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val
> > 2452143445 ecr 0], length 0
> > 11:17:54.318888 IP 192.168.1.51.56576 > 199.15.84.131.43: Flags [S], seq
> > 4226693514, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val
> > 2452146645 ecr 0], length 0
> >
> > Wondering where that IP comes from?  It's org.whois-servers.net, a.k.a.
> > whois.publicinterestregistry.net:
> >
> > $ host org.whois-servers.net.
> > org.whois-servers.net is an alias for whois.publicinterestregistry.net.
> > whois.publicinterestregistry.net has address 199.15.84.131
> > whois.publicinterestregistry.net has IPv6 address 2001:500:106::17:12
> >
> > I don't use IPv6, so I can only confirm IPv4.
> >
> > --
> > | Jeremy Chadwick                                   jdc at koitsu.org |
> > | UNIX Systems Administrator                http://jdc.koitsu.org/ |
> > | Making life hard for others since 1977.             PGP 4BD6C0CB |
> >
> > On Sat, Sep 10, 2016 at 10:53:53AM -0700, Aaron C. de Bruyn via Outages
> > wrote:
> > > I've tried running a few whois queries in the .org domain over the last
> > few
> > > minutes and I'm getting "connect: Network is unreachable".
> > >
> > > I tried from a few websites (Godaddy, namecheap, gkg, etc...) and their
> > web
> > > apps all break, return blank responses, etc...
> > >
> > > Anyone else seeing the same thing?
> > >
> > > -A
> >
> > > _______________________________________________
> > > Outages mailing list
> > > Outages at outages.org
> > > https://puck.nether.net/mailman/listinfo/outages
> >
> >
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages