[Outages-discussion] Mysterious (broken?) mailers and outage list subscriptions
Jeremy Chadwick
jdc at koitsu.org
Sat Mar 1 02:20:00 EST 2014
CC'ing Jared because, well, yeah. Short version is: if rfishler at he.net
is a subscribed member, I would suggest removing them because there is
something fishy going on here.
I fear even sending this just based on what has transpired so far...
So my earlier mail to the list containing a couple attachments, mtr.sh
and a config file, apparently caused a wonderful trickle-down effect
exposing a whole slew of idiocy. Situations like this are never terse,
and it seems like the number of people who understand mail these days
are getting fewer and fewer. Let's start with these I got:
500 + 02/28 21:40 Mailbot for he.net (8.7K) DSN: failed (Re: [Outages-discussion] [outages] Level3 Routing)
501 + 02/28 21:40 Mailbot for he.net (8.5K) DSN: failed (Re: [outages] [Outages-discussion] Level3 Routin)
These DSN bounces came from SMTP server 216.218.186.2 (he.net) with the
following message:
> This is a Delivery Status Notification (DSN).
>
> I was unable to deliver your message to me at staticsafe.ca.
>
> I said
> RCPT TO:<me at staticsafe.ca>
>
> And they gave me the error;
> 550 5.7.1 <me at staticsafe.ca>: Recipient address rejected: Message rejected due to: SPF fail - not authorized. Please see http://www.openspf.net/Why?s=mfrom;id=jdc@koitsu.org;ip=2001:470:0:76::2;r=me@staticsafe.ca
The SPF analysis is correct: my domain, koitsu.org, does not have any
he.net SMTP servers listed as authorised senders for mail originating
from the koitsu.org domain. The only authorised senders of mail for my
domain are 1) my VPS IP address, and 2) all of Comcast's IP space (both
their IPv4 and IPv6 space, though I exclusively only use IPv4 myself,
Comcast's SMTP servers do use IPv6). My VPS sends mail out under that
domain, and because I use Comcast residential services directly, this is
needed (yes I'm aware of the caveats).
What's broken here is that *I* did not send mail through he.net's SMTP
servers directly in any way/shape/form. My residential mail (which this
was) always goes through Comcast's SMTP servers (using SMTP AUTH).
ALWAYS. I have the SMTP transaction logs for the mails in question as
well (Comcast accepting said mails).
What this indicates is that some mailer somewhere is mis-parsing
Received headers (or possibly manipulating/munging mail headers in a
very bad way) such that staticsafe.ca's mail server actually thought I
sent mail through he.net natively. I did no such thing.
So where did these come from? There is some insight below as to who at
he.net may be involved in this madness, but it's impossible for me to
tell because I can't tell what's going on in the midst of it all.
Next idiocy:
I received 2 additional copies of __my own mails__ that I sent to
me at staticsafe.ca + outages at outages.org + outages-discussion at outages.org.
When these arrived I said "Uh, WHAT?". But within them something
becomes obvious...
outages:
697 02/28 21:23 staticsafe (1.0K) | |=>Re: [outages] [Outages-discussion] Level3 Routing...
698 F 02/28 18:34 To staticsafe (4.2K) | `->Re: [outages] [Outages-discussion] Level3 Routing...
699 F 02/28 18:34 To staticsafe (4.5K) | `=>
outages-discussion:
35 02/28 21:23 staticsafe (1.0K) |=>
36 F 02/28 18:34 To staticsafe (4.2K) `->
37 F 02/28 18:34 To staticsafe (4.5K) `=>Re: [Outages-discussion] [outages] Level3 Routing...
The important mails are the ones at the ends (#699 and #37). Those
mails have *significantly* different mail headers in them that are not
my own -- they have, you guessed it, he.net involved. Let's look at
#37:
> Received: from omake.koitsu.org [208.79.90.130]
> by icarus.home.lan with IMAP (fetchmail-6.3.26)
> for <jdc at localhost> (single-drop); Fri, 28 Feb 2014 21:57:03 -0800 (PST)
> Received: from puck.nether.net (puck.nether.net [204.42.254.5])
> by omake.koitsu.org (Postfix) with ESMTP id 4B85F61FB6
> for <jdc at koitsu.org>; Fri, 28 Feb 2014 21:57:02 -0800 (PST)
> Received: from puck.nether.net (localhost [127.0.0.1])
> by puck.nether.net (8.14.8/8.14.5) with ESMTP id s215uhlG027371;
> Sat, 1 Mar 2014 00:57:00 -0500
> Received: from he.net (he.net [IPv6:2001:470:0:76::2])
> by puck.nether.net (8.14.8/8.14.5) with SMTP id s215eGhV023068
> for <outages-discussion at outages.org>; Sat, 1 Mar 2014 00:40:16 -0500
> Received: from he.net ([127.0.0.9]) by he.net for
> <outages-discussion at outages.org>; Fri, 28 Feb 2014 21:39:42 -0800
> Received: from puck.nether.net ([2001:418:3f4::5]) by he.net for
> <rfishler at he.net>; Fri, 28 Feb 2014 18:40:23 -0800
> Received: from puck.nether.net (localhost [127.0.0.1]) by puck.nether.net
> (8.14.8/8.14.5) with ESMTP id s212bewT002737; Fri, 28 Feb 2014 21:40:16
> -0500
> Received: from qmta05.emeryville.ca.mail.comcast.net
> (qmta05.emeryville.ca.mail.comcast.net
> [IPv6:2001:558:fe2d:43:76:96:30:48]) by puck.nether.net (8.14.8/8.14.5)
> with ESMTP id s212YTsi001879 for <outages at outages.org>; Fri,
> 28 Feb 2014 21:34:29 -0500
> Received: from omta03.emeryville.ca.mail.comcast.net ([76.96.30.27]) by
> qmta05.emeryville.ca.mail.comcast.net with comcast id
> Y1rU1n0030b6N64A52aVUs; Sat, 01 Mar 2014 02:34:29 +0000
> Received: from jdc.koitsu.org ([76.102.14.35]) by
> omta03.emeryville.ca.mail.comcast.net with comcast id
> Y2aT1n00a0lNtxY8P2aUFJ; Sat, 01 Mar 2014 02:34:28 +0000
> Received: by icarus.home.lan (Postfix, from userid 1000) id DB3DE73A3B;
> Fri, 28 Feb 2014 18:34:27 -0800 (PST)
Yes, lots to try and follow, but the SMTP flow is as follows:
icarus.home.lan (my home FreeBSD box, sends mail out via Comcast)
-> omta03.emeryville.ca.mail.comcast.net
-> qmta05.emeryville.ca.mail.comcast.net
-> puck.nether.net (via IPv6)
-> puck.nether.net (localhost/itself, likely mailman)
-> he.net (via IPv6) (specifically to user rfishler at he.net)
-> he.net (itself; what is 127.0.0.9? Hmmm...)
-> puck.nether.net (via IPv6)
-> puck.nether.net (localhost/itself, likely mailman)
-> omake.koitsu.org (my VPS)
-> icarus.home.lan (polled omake via IMAP)
It looks to me like something at he.net or "somewhere in the mess that
is HE" (I am not very fond of them, sorry, and I'm even more likely to
lash out when I have to reverse-engineer this stuff) may be doing some
kind of mail redirection, forwarding ("bouncing"), etc. and is doing it
wrong. The term "bounce" is something applicable here -- not "bounce as
in a rejection", but bounce as in "send an exact copy somewhere, the
only difference being the SMTP convo (ex. MAIL FROM/RCPT TO)".
When I see stuff like this I think of broken mailers or badly-written
procmail rules (the latter are incredibly common).
Message #699 is the same way/has the same flow.
Finally, the last batch of idiocy:
503 + 02/28 20:38 Administrator (0.3K) [MailServer Notification]Attachment Blocking Notification
504 + 02/28 20:44 Administrator (0.3K) [MailServer Notification]Attachment Blocking Notification
505 + 02/28 23:56 Administrator (0.3K) [MailServer Notification]Attachment Blocking Notification
506 + 03/01 00:01 Administrator (0.3K) [MailServer Notification]Attachment Blocking Notification
These came from TREND_ON_excaspv01cb at crackerbarrel.com ("From:" line,
not MAIL FROM), specifically SMTP server 170.58.1.25
(barracuda.crackerbarrel.com), complaining about attachments (oh my god,
a file with a .sh extension!):
> The mtr.sh has been blocked,
> and Quarantine entire message has been taken on 2/28/2014 8:38:05 PM.
> Message details:
> Server: EXCASPV01CB
> Sender: jdc at koitsu.org;
> Recipient:
> me at staticsafe.ca;outages at outages.org;outages-discussion at outages.org;
> Subject: Re: [Outages-discussion] [outages] Level3 Routing...
> Attachment name: mtr.sh
Each of the 4 mails had different timestamps in their bodies, and
slightly munged Subject lines to boot:
> and Quarantine entire message has been taken on 2/28/2014 8:43:23 PM.
> Subject: Re: [outages] [Outages-discussion] Level3 Routing...
> and Quarantine entire message has been taken on 2/28/2014 11:56:09 PM.
> Subject: Re: [outages] [Outages-discussion] Level3 Routing...
> and Quarantine entire message has been taken on 3/1/2014 12:00:33 AM.
> Subject: Re: [Outages-discussion] [outages] Level3 Routing...
Wow, the mtr.sh attachment got blocked, probably because the software
being used on barracuda.crackerbarrel.com looked at the extension and
said "oh noez! .sh!! What is that!? TEH SCARIEZ" and decided to announce
that fact. I really didn't need to be told of this -- the person who
probably SHOULD have been told was the recipient that relies on that mail
server.
And why is there 4 of them? Likely because of the weird he.net
situation. Two were likely my original mails, and two were the "copies"
that something at he.net sent (I'm guessing here). The web archive on
puck.nether.net doesn't show these duplicates, but I believe mailman,
given this situation, very likely considered them duplicates.
I would really, really love to know what the hell happened here. The
last incident with the Barracuda crap is one thing, but the stuff
involving he.net is just unacceptable (if this was an IRL convo, I'd use
the phrase "f***ing rude").
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |
More information about the Outages-discussion
mailing list