buryspam
Version 2

Contents

• Introduction
• Requirements
• Upgrading, Installation and Setup
• Using IMAP
• Misclassified Messages
• Displaying Statistics
• Changes from Version 1
• Command line options
• Configuration Parameters: ~/.buryspamrc
• Potential Problems
• Technical Documentation

Introduction

This page describes Version 2 of the buryspam Bayesian spam filter. Like the first version, this filter was influenced by Paul Graham's A Plan for Spam essay from August 2002. Some modifications to the algorithm have been made to help improve the effectiveness of the filter (see the Changes section). Informal testing suggests that this new filter may identify about 70% to 80% of the spam that was missed by the previous version of buryspam. This new version should also be faster, but requires more memory during initialization (about a gigabyte of memory should be sufficient).

The original buryspam script was intended to run be on mail servers via procmail. While Version 2 can still be used in this manner, it also contains a simple IMAP client which allows it to transfer messages via IMAP prior to filtering. See the Using IMAP section for more details. The script is pre-configured to download messages from MUN's IMAP server (mail.mun.ca).

Note that buryspam is not intended to be used directly by mail user agents (MUA) such as pine, Mozilla Thunderbird or Microsoft Outlook. Instead, it is configured to be used on MUN's computer systems as a mail filtering program invoked by procmail. It may also be used as a transfer/filter client which retrieves messages from a remote IMAP server and sorts spam and non-spam messages into predefined mboxes on the local machine. In both cases, an e-mail client, such as pine, can then be used to read the filtered messages.

Requirements

buryspam has the following requirements:

• A relatively recent version of the ruby scripting language must be installed (the current version of buryspam was developed using version 1.8.6).
• If you receive messages via a mail transfer agent (MTA) such as postfix or sendmail then procmail should also be installed on the machine and the MTA should be set up to invoke procmail automatically as messages are received. Modifications to your .procmailrc file will be necessary to call buryspam on received messages. If you intended to receive messages using buryspam's built-in IMAP client instead of an MTA, then there is no need to have procmail installed.
• You must also have a collection of recent spam and non-spam messages for initialization purposes — several hundred of each may be sufficient to get you started. To increase the filter's accuracy, it can be reinitialized later as more spam/non-spam messages are received. The filter requires direct access to these messages for initialization. These messages will typically be stored in the mail directory of your CS account.
• You should be comfortable using the command line and modifying configuration files manually. If you aren't, then buryspam is probably not for you.

Upgrading, Installation and Setup

1. Upgrading

   If upgrading from a previous version of buryspam you should temporarily comment out any rules in your ~/.procmailrc that invoke the buryspam filter by placing a # character at the beginning of the lines. In particular, rules of the form:

   :0fw
   | $HOME/bin/buryspam.rb --filter
   

   should be disabled while the new filter is being setup. This will mean, of course, that spam will not be filtered during setup of the new filter. The lines will be uncommented by a subsequent step.

   It may also be a good idea to make backups of your buryspam.rb and ~/.buryspamrc files and your word file database before continuing so they can be restored if something goes wrong. Note that the word database file and cache files generated by the first version are incompatible with the corresponding files used by the new version of buryspam.

2. Copy the buryspam.rb file to a subdirectory of your home directory.

   The buryspam script currently resides in the ~donald/pub/bin directory. This should be available from any machine with access to the grad partition. You should have your own copy of the script in case the grad partition becomes unavailable for some reason.

   $ cp ~donald/pub/bin/buryspam.rb ~/bin/buryspam.rb
   

3. Separate your spam and non-spam mail
   

   The buryspam script needs a collection of both spam and non-spam messages for initialization purposes. Separate your spam and non-spam messages into different directories. The spam and non-spam may be spread across several directories and subdirectories, but each directory should contain only spam or non-spam messages. Each file containing e-mail messages must be in mbox format. Files that do not appear to be in mbox format will be ignored during initialization.

4. Setup your ~/.buryspamrc configuration file.
   

   At minimum, your ~/.buryspamrc file must set three configuration parameters: bad_dirs and good_dirs, each of which is a list of directories separated by commas that hold the spam and non-spam messages, respectively; and word_file which is the full pathname of the file which will be used to store word probabilities and other information necessary for filtering messages. All directories should be absolute (not relative) and may use the tilde (~) to represent your home directory. e.g.,

   bad_dirs  = "~/mail/junk"
   good_dirs = "/users/cs/grad/donald/mail, ~/mail/old"
   word_file = "~/mail/lib/buryspam.data"
   

   Note that these configuration parameters are the same as the previous version of buryspam. Because other files and subdirectories (e.g., lock files, cache and log files) are stored in the same directory as the word file, the word file should be located in an empty directory. This directory will be created if it did not previously exist.

   Many other configuration parameters are also supported for fine-tuning the filter. Two configuration parameters that you may want to set are missed_spam_file and spam_file.

5. Initialize the filter
   

   To initialize the filter, simply give the command:

   $ ruby ~/bin/buryspam.rb --init
   

   This will scan over the good/bad directories indiciated by the ~/.buryspamrc configuration file and process each of the mboxes in these directories. This may take several minutes to complete. After it is finished, the Bayesian database will be written to the word_file parameter you specified in your configuration file.

   By default, buryspam will use the last three years of good messages and the last one year of bad messages to initialize the filter. If this takes too long to initialize or if buryspam runs out of memory during initialization, the time periods can be shortened by using the bad_init_date_range and/or good_init_date_range configuration parameters in your ~/.buryspamrc file. e.g.,

   # Use the last three months of spam for initialization.
   bad_init_date_range   = 3m
   
   # Use good messages received during the last year for initialization.
   good_init_date_range  = 1y
   

6. Setup your ~/.procmailrc file
   

   In order to execute the buryspam filter, it must be invoked by a rule in your ~/.procmailrc file. You can now uncomment the lines that you commented out in Step 1 if you are upgrading from a previous version of buryspam, but please read the rest of this step to ensure that your .procmailrc file is setup correctly.

   If you are using buryspam for the first time, here is a sample ~/.procmailrc file:

   SHELL=/bin/sh
   PATH=/bin:/usr/bin
   MAILDIR=$HOME/mail
   LOGFILE=$MAILDIR/.procmail.log
   
   # Replace 'username' with your login name.
   :0fw:/tmp/username.lock
   | $HOME/bin/buryspam.rb --filter
   
   :0:
   * ^X-Buryspam-Spam: Yes
   junk/spam
   

   Replace username with your login name (or some other text for the lockfile name). Also make sure that you specify the correct path of your copy of the buryspam.rb script.

   This file contains two rules. The first rule calls the buryspam filter which will place a X-Buryspam-Spam header line on messages as part of the filtering process. The second rule uses this header line to determine if the message should be stored in the spam folder of the junk directory. The junk directory is relative to the MAILDIR directory specified near the beginning of the ~/.procmailrc file and must already exist. The directory that stores all your spam (~/mail/junk in this case) should be one of the directories specified by the bad_dirs configuration parameter. A .procmail.log file is created in your MAILDIR directory by procmail. This file may contain helpful diagnostic messages if buryspam fails for some reason. Other rules to automatically sort messages from other sources (e.g., mailing lists, NEWSLINE announcements etc.) can also be added to your .procmailrc file.

   Note: The previous version of buryspam added a X-Bayesian-Spam header to messages. This has been replaced with the X-Buryspam-Spam header line in version 2.

At this point, all messages that you receive should pass through the buryspam filter. Any spam messages should be automatically deposited in your mail/junk/spam file. The rest of your messages should be stored according the the remainder of the rules in your .procmailrc file.

Using IMAP

If your messages are stored on a remote mail server, buryspam can still be used to transfer and filter these messages. The messages are converted to mbox format and stored in a file on your local file system. Note that after a message has been successfully transferred/filtered, it is deleted from the IMAP server. To transfer and filter messages from an IMAP server, use the --transfer command line option:

$ ruby buryspam.rb --transfer

After prompting you for your IMAP password, this will connect to the configured IMAP server and call procmail to process and store each message as they are transferred from the IMAP server. If your ~/.procmailrc file is set up as specified in the previous section, then this will result in another invocation of a buryspam process to filter each message transferred. Because two processes are created per message, this method of transferring/filtering messages is relatively slow. This should not be a problem if you transfer messages regularly, thereby preventing a large number of messages from accumulating on the server.

To automatically transfer messages at a regular time interval, the --poll command line option can be used:

$ ruby buryspam.rb --poll

This is identical to the --transfer option except that instead of terminating after transferring and filter messages, buryspam will remain running in the background (as a daemon) and periodically connect to the IMAP server to transfer messages and filter them using procmail. See the poll_timer configuration variable for information on how to set the time interval.

If you want to transfer several hundred or more messages from the IMAP server, then the --bulk command line option will likely be considerably faster:

$ ruby buryspam.rb --bulk

This will transfer and filter the messages from the IMAP server within a single buryspam process. All non-spam messages will be saved to your spool file as indicated by the mail_spool configuration parameter and all spam will be saved to the file specified by the spam_file configuration variable. Because your ~/.procmailrc file is ignored, no other sorting of messages will occur. This option is recommended if you don't have a .procmailrc file or if you have a very simple .procmailrc file which does no other sorting of messages beyond depositing spam into a spam folder.

Bulk filtering will be used if procmail is not found on your system by the configuration, even if the --transfer or --poll options are given on the command line.

There are several configuration parameters related to transferring and filtering messages via IMAP, including: imap_client, imap_inbox, imap_port, imap_server, imap_use_ssl, imap_username, mail_spool, poll_timer, procmail and spam_file

Misclassified Messages

False Positives

As the filter is used, false positives (non-spam messages incorrectly identified as spam) may occur, but they should be very rare. If you haven't received a message you were expecting, then the message may have been identified as a spam message and sent to your spam folder. Check the spam files in the directories specified by the bad_dirs configuration parameter. You can use the --grep option of buryspam to search spam files for regular expressions to help making finding false positives easier: e.g.,

$ ruby buryspam.rb --grep '/Name|Thing/' spam* |less

Non-spam messages that have been incorrectly identified as spam should be moved out of the spam directory and into one of the good_dirs directories. Note that spam files are automatically renamed before they grow beyond a certain size (see the spam_file_size configuration parameter), so there may be many spam files.

False Negatives

If the spam filter incorrectly identifies a spam message as non-spam (false negative), don't delete the message. Instead, either save it to a file in the spam messages directory or save it to a file which will be moved to a spam message directory during a subsequent initialization of the filter with --init. (see the missed_spam_file configuration parameter). If the filter is repeatedly misclassifying messages that have the same content, then the filter should probably be reinitialized.

Leaving spam messages in a directory specified in the good_dirs directory list or leaving non-spam messages in a directory specified by the bad_dirs configuration parameter may decrease the effectiveness of the filter the next time it is initialized.

If the filter seems to be generating a lot of false positives or false negatives, then it is possible that you may have initialized the filter with some spam and/or non-spam messages in the wrong directories. Turn on the verbose_hdr configuration parameter and wait for subsequent misclassifications to occur. Then study the X-Buryspam-Words: header lines of messages which have been misclassified to see what discriminating words (and probabilities) the filter used to classify the message. If a lot of words with high probabilities are appearing in your non-spam messages or if a lot words with low probabilities are appearing in your spam messages, then you can --grep for these words in the messages that were used to initialize the filter to determine if your initialization corpus was tainted.

Troubleshooting and Log files.

If there are any problems, you should check your ~/mail/.procmail.log file (specified in the .procmailrc file, above). You can also check the log files in the buryspam log directory. Searching for the log files for lines that begin with E may be helpful.

By default, the log directory is named log and is located in the same directory as the word_file. More information can be generated in the log file by setting the log_level configuration variable to debug.

If buryspam encounters a message that it has trouble processing from the IMAP server, any messages following the troublesome message may not get processed. As a result, buryspam will become wedged and will not download any messages from the IMAP server. To fix this problem, it may be necessary to login to the IMAP server directly (via a webmail interface, for example) and delete the message that is causing the problem.

Displaying Statistics

Various statistics about the rate and amount of spam you receive as well as the filter's effectiveness in blocking it can be obtained using the --stats option.

$ ruby buryspam.rb --stats

Buryspam v2.0.0 (hostname)
Statistics...

Last reinitialized:   2008-10-27 20:08:52 (3.8 days ago)
Unprocessed messages: 0
Total spams:          3657
Spam frequency:       971.2 spams/day
Spam period:          89.0 secs/spam
False negatives:      13
False negatives/day:  3.5
Accuracy:             99.64%

If it has been a long time since the filter was initialized, then this command may take a while to run — be patient.

Changes from Version 1

Although buryspam has been completely rewritten, its basic usage hasn't been changed considerably in Version 2. The filter is initialized and used the same way as the previous version. Internally, the code is tighter and in some cases faster than the previous version. Some of the major changes include:

• When selecting discriminating word samples during filtering, proportional subsampling is used when the number of samples exceeds the sample size. This should reduce poisoned spam messages that get through the filter by including a lot of random words.
• The filter maintains lists of IP address from which non-spam and spam messages are received. When a new message is received its octets are compared with the octets of IP address from which spam was received. Each matching octet causes a good word to be shifted out of the discriminating word samples and a fictional bad "word" to be introduced into the word samples.
• There are now two cache files for each mbox. Also, the cache files are comprised of serialized hashes and store more information. This should speed up initialization.
• In order to conserve memory, the previous version of buryspam would only load parts of an mbox into memory at one time during initialization. This version loads entire mboxes into memory as they are processed. This simplifies the code considerably but requires more memory than the previous version of buryspam.
• In the previous version, new messages had to be filtered by calling buryspam from a rule in the .promailrc file. Now, messages can be retrieved from an IMAP server and subsequenty filtered. Periodically polling for messages on an IMAP server can also be performed by running buryspam as a daemon. See the Using IMAP section for more details.
• Configuration parameters and command line options of questionable usefulness have been eliminated, particularly those related to HTML message parsing and to automated testing and report generation.
• A rotating log file (provided by ruby's Logger library) is now used which records actions taken by buryspam as it runs. This log file may be useful in helping to diagnose problems encountered by buryspam. New configuration parameters related to logging are: log_file, log_file_count, log_file_size and log_level
• More comprehensive help is available with the --help command line option. When given a command line option or a configuration parameter as an argument to the --help option, documentation related to that command line option or configuration parameter is displayed. For example:

  $ ruby burypam.rb --help word_file
  

  will display help on the word_file configuration parameter. HTML documentation for all the command line options and configuration parameters can also be generated.
• This version of buryspam now makes use of lockfiles to prevent unsynchronized access to files and to prevent two related buryspam processes from being run simultaneously.
• The probabilities assigned to words has their precision limited to three digits after the decimal, by default; but this can be changed via the precision parameter.

Command line options

The available command line options are displayed in the usage message which is provided when the script is executed without any command line options.

$ ruby buryspam.rb

Buryspam v2.2.1 (garfield)
Help...

Usage: buryspam.rb <option> [<mboxes>]

Available command line option modes:

--init                 Initialize the word probability and IP address database.
--filter               Filter mboxes, testing for spam.
--transfer             Transfer/filter mail from IMAP server using procmail.
--poll                 Do --transfer periodically.
--bulk                 Transfer/filter mail from IMAP server without procmail.
--decode               Decode messages in mboxes (i.e., base64/qp encodings).
--grep '/pattern/'     Display messages that contain 'pattern' (implies decode).
--colour               Assign probability colours (in HTML) to messages' words.
--stats                Display various statistics regarding filter performance.
--log [severity]       Display log messages with specified severity (d,i,w,e,f).
--help [option|param]  Display this help message or display help for the
                       specified command line option or configuration parameter.

For --filter, --decode and --grep, mbox input is taken from either stdin
or from command line filenames.  The other modes do not use mboxes.
NOTE: Only one option mode can be specified at a time.

Other options:

--override <params>    Override configuration parameters.

--override can be used in conjunction with the other modes above.
<params> is string of the form: 'param = value; param = value; ...'
Run 'ruby buryspam.rb --help override' for more details

A more detailed description of each of the command line options is given below:

Configuration Parameters: ~/.buryspamrc

Configuration parameters are stored in the ~/.buryspamrc configuration file which is read during startup. The syntax for each line is:

parameter_name = value

Space and tabs are ignored during parsing. Lines beginning with '#' are ignored. Leading and trailing quotes around the value are ignored. The buryspam.rb script will terminate prematurely if there are any errors in the ~/.buryspamrc file. Configuration parameters may be overridden on the command line with the --override command line option. A complete description of all available configuration parameters is given below. Note that the word_file, bad_dirs and good_dirs configuration parameters must be specified.

Potential Problems

In the interest of full disclosure, here are some issues associated with buryspam:

• During --poll mode, the IMAP password is always stored in memory. This may present a security risk. Note that the password is never intentionally written to disk (swap partitions notwithstanding).
• If buryspam dies while polling or if the machine on which it is running is restarted, then no incoming mail will be received until buryspam is restarted with the --poll option.
• Because of the use of lock files, there is a possibility of deadlock occuring. Hopefully, that possibility should be very remote. If the word_file cannot be saved during initialization due to a timeout waiting for a lock and no other buryspam processes are running, then it is possible that an errant lock may still be held on the word file. Renaming or removing the word file and then re-initializing should correct the problem (but make sure that no other buryspam process will attempt to access the missing word file while it is being regenerated.)
• Log files may roll over a little early when more than one buryspam process is run. As a result, a log file will sometimes be rolled over before it reaches the size specified by the log_file_size configuration parameter. Also, two concurrent buryspam processes may step on each other's log entries.
• The memory requirements for buryspam are a bit excessive during initialization and filtering. Having multiple copies of buryspam running on a single machine simultaneously may tax its memory resources.
• It is not possible to run two transfer related buryspam processes simultaneously. This means, for example, that if you are running a buryspam polling daemon, you cannot subsequently run another polling daemon or invoke buryspam using --transfer or --bulk. This is to prevent the competing IMAP clients from putting the IMAP server's inbox in an inconsistent state. Ideally, concurrent buryspam transfer processes should be permitted if they are accessing different mailboxes and/or different IMAP servers.
• After transferring messages from an IMAP server, a Connection reset by peer during disconnect (warning ignored) warning may sometimes occurs. This only appears to happen when connecting to garfield's IMAP server from garfield itself. I'm not sure what the ramifications of this are — at worst, it seems, messages may be left unexpunged from the IMAP mail server. Running a transfer mode (--transfer, --poll or --bulk) on the IMAP server itself is therefore prohibited.

Technical Documentation

Technical documentation is available in rdoc format. (This documentation is intended for those who wish to modify the buryspam code and will likely not be useful to end users.)