Return-Path: gward@python.net
Delivery-Date: Mon Sep 9 22:01:10 2002
From: gward@python.net (Greg Ward)
Date: Mon, 9 Sep 2002 17:01:10 -0400
Subject: [Spambayes] python.org email harvesting ready to roll
Message-ID: <20020909210110.GA2224@cthulhu.gerg.ca>
[followups to spambayes@python.org please, unless you're specifically
concerned about some particular bit of email policy for python.org]
OK, after much fiddling with and tweaking of /etc/exim/exim4.conf and
/etc/exim/local_scan.py on mail.python.org, I am fairly confident that
I can start harvesting all incoming email at a moment's notice. For the
record, here's how it all works:
* exim4.conf works almost exactly the same as before if the file
/etc/exim/harvest does not exist. That is, any "junk mail
condition" that can be detected by Exim ACLs (access control lists)
is handled entirely in exim4.conf: the message is rejected before it
ever gets to local_scan.py. This covers such diverse cases as
"message from known spammer" (reject after every RCPT TO command),
"no message-id header", and "8-bit chars in subject" (both rejected
after the message headers/body are read).
The main things I have changed in the absence of /etc/exim/harvest
are:
- don't check for 8-bit chars in "From" header -- the vast
majority of hits for this test were bounces from some
Asian ISP; the remaining hits should be handled by SpamAssassin
- do header sender verification (ie. ensure that there's a
verifiable email address in at least one of "From", "Reply-to",
and "Sender") as late as possible, because it requires DNS
lookups which can be slow (and can also make messages that
should have been rejected merely be deferred, if those DNS
lookups timeout)
* if /etc/exim/harvest exists, then the behaviour of all of those
ACLs in exim4.conf suddenly changes: instead of rejecting recipients
or messages, they add an X-reject header to the message. This
header is purely for internal use; it records the name of the folder
to which the rejected message should be saved, and also gives the
SMTP error message which should ultimately be used to reject
the message.
Thus, those messages will now be seen by local_scan.py, which now
looks for the X-reject header. If found, it uses the folder name
specified there to save the message, and then rejects it with the
SMTP error message also given in X-reject. (Currently X-reject is
retained in saved messages.)
If a message was not tagged with X-reject, then local_scan.py
runs the usual virus and spam checks. (Namely, my homebrew
scan for attachments with filenames that look like Windows
executables, and a run through SpamAssassin.) The logic is
basically this:
if virus:
folder = "virus"
else:
run through SpamAssassin
if score >= 10.0:
folder = "rejected-spam"
elif score >= 5.0:
folder = "caught-spam"
Finally, local_scan.py writes the message to the designated folder.
By far the biggest folder will be "accepted" -- the server handles
2000-5000 incoming messages per day, of which maybe 100-500 are junk
mail. (Oops, just realized I haven't written the code that actually
saves the message -- d'ohh! Also haven't written anything to
discriminate personal email, which I must do. Sigh.)
* finally, the big catch: waiting until after you've read the message
headers and body to actually reject the message is problematic,
because certain broken MTAs (including those used by some spammers)
don't consider a 5xx after DATA as a permanent error, but keep
retrying. D'ohh. This is a minor annoyance currently, where a fair
amount of stuff is rejected at RCPT TO time. But in harvest mode,
*everything* (with the exception of people probing for open relays)
will be rejected at DATA time. So I have cooked up something called
the ASBL, or automated sender blacklist. This is just a Berkeley DB
file that maps (sender_ip, sender_address) to an expiry time. When
local_scan() rejects a message from (sender_ip, sender_address) --
for whatever reason, including finding an X-reject header added by
an ACL in exim4.conf -- it adds a record to the ASBL, with an expiry
time 3 hours in the future. Meanwhile, there's an ACL in exim4.conf
that checks for records in the ASBL; if there's a record for the
current (sender_ip, sender_address) that hasn't expired yet, we
reject all recipients without ever looking at the message headers or
body.
The downside of this from the point-of-view of corpus collection is
that if some jerk is busily spamming *@python.org, one SMTP
connection per address, we will most likely only get one copy. This
is a win if you're just thinking about reducing server load and
bandwidth, but I'm not sure if it's helpful for training spam
detectors. Tim?
Happy harvesting --
Greg
--
Greg Ward <gward@python.net> http://www.gerg.ca/
Budget's in the red? Let's tax religion!
-- Dead Kennedys