------------ INSTALLATION ------------ SPASTIC can be installed or upgraded using the "setup" script included in the package. Manual installation instructions are also available in case you prefer to install or upgrade yourself. Also, if you are using spastic for the first time, please review the manual instructions for tips on what files to modify AFTER the installation. Note: If you are upgrading and want to upgrade manually, see the "Upgrading from a previous version" section near the bottom of this file. If you are upgrading from a version < 2.4 AND you use the the filtering in your mail user agent (evolution, kmail, mozilla mail, etc.), please update your e-mail filters to check for spam using the new X-Spam-Status header instead of checking the Subject header. Starting with 2.4, spastic adds an X-Spam-Status header instead of changing the Subject header. For more information, see "Upgrading from a previous version" section near the bottom of this file, and read item 8. Note to qmail users: see README.qmail Note to evolution/kmail/other filtering MUAs users: see README.alternative Note to pine users: see README.pine Note: If you want to use procmail/spastic for global filtering of ALL e-mail for all users, see the INSTALL.MTA file. ------------ Requirements ------------ SPASTIC was designed to work using basic Linux and Unix-like OS utilities. In most cases, all requirements for proper operation are already available on your system. 1. procmail* 2. grep/fgrep/egrep 3. formail 4. MTA (postfix, sendmail, qmail, exim, etc.) 5. dig (for optional domain check) 6. sed and awk (for optional rotate-spam script) *SPASTIC works in conjunction with (and requires) Procmail, the mail processing utility for *nix. Among Procmail's many uses are its ability to sort and pre-process your incoming mail. To find out whether you have procmail already installed on your system (and if so, its location), do user@world:~>which procmail To find out which version you have installed, do user@world:~>procmail -v If you'd like more information about procmail and its wonderful capabilities, or for the latest version of Procmail, visit the Procmail homepage at http://www.procmail.org/. At the time of writing the most current version is 3.22. Or try these commands: user@world:~>man procmail user@world:~>man procmailex user@world:~>man procmailrc ----------------------------------------------- Automated installation with the "setup" script: ----------------------------------------------- 1. Copy the spastic-x.x-tar.gz file to your home directory or another work directory and untar/ungzip it with: tar xzvf spastic-x.x-tar.gz 2. Change to the newly-unpacked spastic directory and run "./setup". 3. If you did not receive any errors, continue reading below for information on which files to tweak and to learn about optional features. The setup script makes a backup copy of any previous spastic directory it can find to $HOME/.spastic.old. It also backups up the .procmailrc to .procmailrc.old. Then, it installs the new version in $HOME/.spastic and copies over the old filter files, whitelist, allowed, and spam files to the new installation directory, $HOME/.spastic. If you have customized your .procmailrc, then you will need to edit it to apply any custom changes to it. Finally, if you are using the rotate-spam script, it also needs to be updated. -------------------- Manual installation: -------------------- These procedures assume that $HOME/Mail is going to be your mail directory (MAILDIR) and $HOME/.spastic is the spastic directory. Make adjustments if your mail directory is going to be stored elsewhere. 1. Ungzip/untar spastic-x.x.tar.gz from your home directory and rename it to $HOME/.spastic (replace x.x with the current version). To accomplish this, do: user@world:~>tar xzvf spastic-x.x.tar.gz user@world:~>mv spastic $HOME/.spastic Edit the dot.procmailrc file provided. Make sure to note the PATH of procmail, grep, fgrep, egrep, dig, and formail. The default PATH is set to /bin:/usr/bin which should work on most systems. If one of your programs is somewhere else, add the location to the PATH so it will be found when spastic runs. You can find the paths of these programs with "which or "type ", or "locate ". Rename the dot.procmailrc to .procmailrc and move to your home directory. Make sure the MAILDIR (mail directory) and DEFAULT (mailbox) variables are pointed to the right places: MAILDIR is initially set to $HOME/Mail DEFAULT is initially set to $MAILDIR/mbox 2. Rename dot.forward to .forward and move to your home directory. Make sure the path to procmail is correct (defaults to /usr/bin/procmail). 3. Check over each filter file to see if there are any additions or subtractions you'd like to make to the filter lists. The filters are fairly lax to begin with but you can customize them to your heart's content. Make sure you don't have any blank lines in ANY of the filter files. Blank lines can cause valid e-mails to be flagged as spam. reject-body (body of the message) reject-from (From: e-mail header) reject-mailer (X-Mailer: e-mail header) reject-msgid (Message-ID: e-mail header) reject-replyto (Reply-To: e-mail header) reject-subject (Subject: e-mail header) reject-to (To: e-mail header) reject-type (Content-Type: e-mail header) 4. Edit the whitelist file and add the FULL e-mail addresses of friendly people or places, one per line. E-mails arriving with these addresses are delivered immediately and bypass all filtering. The whitelist matches on the To:, From:, and Cc: headers. 5. Send a few test e-mails to yourself, see how it works out. Take a look at the spam file every now and then to make sure no legit e-mail was accidently filtered. Adjust accordingly. If you have problems and you are using Kmail, evolution or similar MUA's (Mail User Agents, or simply e-mail programs,) please see README.alternative.txt. If you still have problems after reading that, or if things just don't seem to work properly, please read FAQ.txt which has very specific procedures which help you identify and solve the most common problems. Note on filters: The filters are set up to scan for fixed length text strings in the header or body of each message. They are NOT case sensitive. You can change the switches on egrep if you want to make them case sensitive. If you understand regular expressions, you can also use them in the filter files for very sophisticated matching. Note on spam destinations: The dot.procmailrc example defaults to sending caught spam to the $HOME/.spastic/spam file. You can review it at your leisure and delete it permanently by clearing the file. There are other things you might want to do with your spam. You could use the $SHREDDER variable as a destination (/dev/null) and e-mail identified as spam will just evaporate. Another thing you can do with captured spam is flag it for filtering by another program. Rather than send captured spam to a specific folder, you can put a special label on it and let it to go to your inbox. Then, using the built-in filtering capabilities of your MUA, filter the e-mails labeled as spam, taking any action you wish. The reason you might want to do it this way is in case your mail program tracks mail differently or stores it in a non-standard format. For example, evolution tracks mail internally in a way that does not allow procmail to safely deliver it directly to the evolution mailboxes. -------------------------- OPTIONAL long reject files -------------------------- In the $HOME/.spastic/reject-long directory are two files that have extra entries already set up for reject-body and reject-subject. These are long lists from the original SPAST project. If you want to be more aggressive with your filters, just copy these files into the $HOME/.spastic directory overwriting the same files that have a smaller number of entries. ---------------------------- OPTIONAL Content-Type filter ---------------------------- While this feature is optional, my experience is that a large percentage of recent spam comes as HTML e-mail. My suggestion is that you add any user or source that you expect to send HTML e-mail to your whitelist, then enable HTML filtering using the reject-type file. Future versions of SPASTIC may have HTML filtering active in the default configuration. The $HOME/.spastic/reject-type file is initially empty. The recipe that uses it will not filter anything until entries are made in the reject-type file. If you want to filter e-mail based on the MIME Content-Type header, you can use this file. The most common use for this is to flag HTML mail as spam. Note that some legitimate mail can be delivered in HTML format, so you will need to account for this in the whitelist or some other way if you want to flag all HTML mail as spam. HTML mail usually comes as one of these types: text/html multipart/mixed multipart/related multipart/alternative Use these in your reject-type file if you want to filter out HTML mail. Add each content-type you want to trap on a separate line in the $HOME/.spastic/reject-type file. As with the other reject files, make sure there are no blank lines in the file. Here are all the valid e-mail MIME Content-Types defined at the time this was written in RFCs 2045-2047: text/plain text/enriched message/rfc822 message/partial message/external-body image/jpeg image/gif video/mpeg audio/basic application/postscript application/octet-stream multipart/mixed multipart/parallel multipart/alternative multipart/digest Here is a non-standard type that some vendors use in their mail programs: text/html The seven primary MIME types are fixed. However, new subtypes change frequently when vendors want to extend the standard. For example, the multipart/related type was defined in another RFC and modified by subsequent RFCs. It takes a lot of work to stay on top of all submitted RFCs. It is not possible for this document to track all MIME changes, just be aware that new subtypes are likely to appear in the future. --------------------------- OPTIONAL rotate-spam script --------------------------- Requirements: cron The rotate-spam script is a BASH script that will copy the spam file to the spam-backups folder using the logrotate style (spam.1, spam.2). This is a change from version 2.2 which named backup spam file with the current date as the name of the file. Another change is that the script now relies on the default cron behaviour of e-mailing the output to you instead of calling the mail program directly. If you are sending stdout and stderr to /dev/null in your cron job, please change it so that the report (which goes to stdout) will get e-mailed to you. The rotate-spam script, is run with no parameters also creates a detailed report of the spam eaten. Here is a sample (short format) report: SPASTIC REPORT SPASTIC ate 345 e-mails for the following reasons: 232 Rejected by Content-Type 64 Not allowed destination 17 Rejected by From 12 Rejected by Subject 8 Rejected by Body 7 From address in a bad format 4 Null Subject 1 No MX record for domain END OF SPASTIC REPORT There are several command line options that can be used. Here is a summary of the syntax: rotate-spam [-s|--short][-r|--report][-l[n]|--log-rotate [n]]|[-q|--quiet] [-s|--short] produces a short report without From line details [-r|--report] produces only a report without rotating the spam file [-l[n]|--log-rotate [n]] changes the number of backup files to keep in the spam-backups directory (the default is 4) [-q|--quiet] rotates the spam files but does not produce a report Options can be combined. Here are some examples: To create a short version of the report and keep 7 backup spam files, use: ./rotate-spam -s --log-rotate=7 To only produce the report, but not rotate files, use: ./rotate-spam --report The rotate-spam script has been updated to retrieve as much information as possible from it's environment. If you are running on Linux, it is unlikely that you will need to modify it at all. However, on other *nix systems, the environment may differ and you may need to set the USER and HOME variables at the top of the script. Make sure the script has the executable permission set. It is designed to be run automatically using cron. To set it up to run automatically, type: crontab -e This will open your default text editor to edit your personal crontab file. For a daily report, create an entry similar to this. What you want is the full path to the rotate-spam script wherever it lives: # Rotate spam file every night at 23:59 every day 59 23 * * * /home/foo/.spastic/rotate-spam For a weekly report, create an entry similar to this: # Rotate spam file every week at 23:59 every Sunday 59 23 * 7 * /home/foo/.spastic/rotate-spam Then, save your changes. For more information on cron, try: man cron man crontab --------------------- Optional allowed file --------------------- The allowed file is initially empty. As long as it is empty (0 bytes), it will NOT be used in the spam filters. However, it provides another way for you to restrict what e-mail makes it into your default mailbox. To use the allowed file, simply list valid e-mail addresses that are allowed as To: or Cc: recipients, one per line. The recipe checks the To: and Cc: headers against the allowed list. If any of the headers match the allowed list, then the e-mail continues on through the rest of the spam checks, otherwise it is dumped into the spam file. The test is NOT case sensitive. IMPORTANT: Include ALL of your e-mail addesses here so you don't miss things that are Cc'ed to you. Also include any mailing lists you belong to and any other address that might be in the To: or Cc: header fields. Of course, if you have your mailing lists in the whitelist, you do not need them in here as well. WARNING: If someone not in your allowed file sends you an e-mail as a blind carbon copy, the Cc and Bcc headers may be stripped and you can't test for it. These e-mails may be flagged as spam. This is a trade-off for the extra protection you get from only allowing certain mail addresses to deliver to you. Be careful! Watch your spam file carefully after you activate this feature. Normally, the e-mail addresses of mailing lists, online newsletters, notifications, or other valid e-mail you might receive that is addressed to someone other than you should go here. If you use it, be sure to list EVERY address that is allowed. Here is an example of entries in the allowed file: me@myaddr1.com me@myaddr2.com me@yetanother.org suse-linux-e@suse.com hardcore-coders@regexp.org ------------------------ Optional domain checking ------------------------ Near the end of the dot.procmailrc file is an Advanced section that includes a domain check rule. It harvests the From, Reply-To (if it exists), and Return-Path (if it exists) addresses and uses the dig program to check for a valid domain mail service record (MX). If the query fails, the e-mail is flagged as spam. Using this test can take extra time and network bandwidth, especially if you receive a lot of e-mail. If you want to use this recipe, just uncomment the lines in the Advanced section. Normally, this kind of test should be done by the mail server that first receives the e-mail (the Mail Transfer Agent). However, most people do not have the pleasure (or pain) or running an Internet mail server and receive their e-mail from an ISP. In this case, the mail has already been delivered and this at least provides a way to check for valid domains before it makes it to your mailbox. If you are setting up SPASTIC as a global filter for a site, please check to see if you can implement this feature in the mail server instead of after the fact in SPASTIC. Most major MTAs that I am aware of have this capability. -------------------------- Optional anti-virus filter -------------------------- Near the end of the Advanced section of the dot.procmail file are two recipes that can flag executable content to protect Windows users. If you are using a *nix desktop, this is not necessary. You would use this filter where SPASTIC was set up as a global spam filter to a network of Windows users and you also wanted to trap incoming executable content and attachments. To use the recipes, simply uncomment them. By default, executable content is just flagged like a normal spam. You may instead want to send them to a local *nix mailbox for analysis before forwarding them to their final destination. This would require modifying the destination in the recipes. --------------------------------- Upgrading from a previous version --------------------------------- If you are upgrading from a previous version of SPASTIC, you can follow these procedures. At long last, there is a script that can do all of this for you called "setup". The setup script is able to upgrade any version after 2.0 if it is located in either $HOME/Mail/spastic or $HOME/.spastic. See more details on the setup script at the top of this file. Manual upgrade instructions: 1. Rename the old /.spastic directory to /.spastic.old to preserve your reject and log files. 2. Untar/ungzip the new spastic-x.x.tar.gz and rename the directory to $HOME/.spastic. 3. Copy any old reject files you have customized, log files you want to keep, and your whitelist and allowed files from the spastic.old directory to the new spastic directory. 4. Delete the spastic.old directory (or not). 5. Copy your $HOME/.procmailrc to $HOME/.procmailrc.old 6. Copy the new $HOME/.spastic/dot.procmailrc to $HOME/.procmailrc. 7. Add your custom changes from the $HOME/.procmailrc.old into the new .procmailrc. 8. If you upgraded from a version < 2.4 AND you use the alternative procmailrc (you do the final filter in your e-mail program), you MUST update your filters to check the "X-Spam-Status" header for a value of Yes. In versions < 2.4, spastic changed the Subject header to "SPAM" and provided a reason. However, starting with version 2.4, the Subject is untouched and a new header is added. Here is what the new header will look like: X-Spam-Status: Yes, Reason: Too many spaces in Subject Your e-mail filters need to check for "X-Spam-Status" = Yes. Note: The reject file names changed from version 1.7 to 2.0 so you will need to rename the old files to the new names if you are upgrading from a version prior to 2.0.