contact :: legal :: webstats  also on this site: - select - home tech writings green business studies conspiracy popular pages welcome  also in this section:  tech index  most popular pages  search: Like this site? Your donation is welcome!

This is the Filter of Last Resort. Reason being, it's very aggressive. You'll see why below. For this reason, this filter should be used last, after all your other email filters. This way, this filter will only ever deal with the dregs, which means it's not so dangerous.

As this is the Filter of Last Resort, it does NOT include filtering for all kinds of spam. Rather, it is designed to kill the spams that the other filters miss.

It is dangerous. Most filters mark the spam and let it be. This filter kills it. No Deleted Items, no Recycle Bin, no undo, no 'are you sure'. Bang bang, dead dead. All that is left is a logfile entry.

Note, the below is a blueprint, there is no download. I do have the software, but:

• it's not fit for public use
• I've currently stopped using it, as I want to keep the evaders, so I can use them to refine Regexfilter's ruleset
• due to these refinements, and other customisations to Regexfilter's ruleset, I have less evaders in any case
• it is a bit rough - certainly it's faster/more efficient to get the regexes sorted, than to whack a bunch of stuff that should have been caught anyway...

automatic whitelist maintenance

To minimise the chances of a legitimate mail being terminated, this filter includes a "make whitelist" command. This command tells the filter to collect all email addresses from inside a number of other files (my address books), eliminate the duplicates and save the list to disk. This list is then used by the "move whitelisted" command, which moves any message containing whitelisted strings to a separate folder (the "whitebox").

This filter also supports three other whitelists, these are good_senders, good_recipients and good_subjects. Any mail containing a whitelisted string in the correct location is automatically "whiteboxed" (moved to the whitebox).

how it works

This filter is comprised of 11 special-purpose tests. Any mail matching one of these tests is deleted. The filters are as follows:

1. missing_addressee (missing 'To:' or 'for' field)
2. missing_sender (missing 'From:' field)
3. unlikely_chars (non-alphabetic subject or sender)
4. unlikely_dates (message date too old, or in future)
5. bounces (mail delivery failure, etc)
6. blacklisted (bad_senders/bad_recipients/bad_subjects)
7. gifs_attached (message has an attached GIF image)
8. X-RBL (message contains X-RBL-Warning: headerline)
9. X-DNS (message contains X-DNS-Warning: headerline)
10. X-SVF (message contains X-Sender-Verification-Failed: headerline)
11. analyse_received (Received: line invalid - see below)

These tests are fairly self-explanatory, with the exception of the analyse_received test. This test analyses the significant Received: headerline inside each mail (there are usually several Received: lines, but only one is relevant for our purpose). Any mail with an invalid Received: line is deleted. The tests for validity are as follows:

1. IP_missing
2. IP_obfuscation
3. IP_unreversible
4. by-line_not_present
5. sending_SMTP_server_unresolvable
6. sending_hostname_not_provided

If these tests all pass, the message is then tested for a mismatch between the sender's hostname and the hostname of the sender recorded by the receiver. Again, a fail results in the message being deleted.

why it works

Spammers try and get their messages through by hiding, disguising or armouring their spams. This filter spends most of its time looking for evidence of armour. It assumes that an attempt at armouring means the mail is spam.

Note that this approach is very unforgiving toward badly configured, but legitimate systems, or systems using non-standard data formats. Another reason to run this filter last. Mistakes can be minimised by keeping the whitelists up-to-date, and encouraging all to run RFC-compliant nodes.

related articles:  email filtering with Spampal  fixing Regexfilter's ruleset  customising Regexfilter's ruleset  refining Regexfilter's ruleset  integrating Spampal with Thunderbird  the Filter of Last Resort  the phish filter