spamoracle - Online in the Cloud

PROGRAM:

NAME

spamoracle - a spam classification tool

SYNOPSIS

spamoracle [-config conf] [-f database] mark [ mailbox ... ]

spamoracle [-config conf] [-f database] add [-v] -spam spambox ... -good goodbox ...

spamoracle [-config conf] [-f database] test [-min prob] [-max prob] [ mailbox ... ]

spamoracle [-config conf] [-f database] stat [ mailbox ... ]

spamoracle [-config conf] [-f database] list regexp ...

spamoracle [-config conf] [-f database] backup > backupfile

spamoracle [-config conf] [-f database] restore < backupfile

spamoracle [-config conf] [-f database] words [ mailbox ... ]

DESCRIPTION

SpamOracle is a tool to help detect and filter away "spam" (unsolicited commercial e-
mail). It proceeds by statistical analysis of the words that appear in the e-mail,
comparing the frequencies of words with those found in a user-provided corpus of known
spam and known legitimate e-mail. The classification algorithm is based on Bayes'
formula, and is described in Paul Graham's paper, A plan for spam,
http://www.paulgraham.com/spam.html.

This program is designed to work in conjunction with procmail(1). The result of the
analysis is output as an additional message header X-Spam: followed by yes, no or unknown,
plus additional details. A procmail rule can then test this X-Spam: header and deliver
the e-mail to the appropriate mailbox.

In addition, SpamOracle also analyses MIME attachments, extracting relevant information
such as MIME type, character encoding and attached file name, and summarizing them in an
additional X-Attachments: header. This allows procmail to easily reject e-mails
containing suspicious attachments, e.g. Windows executables which often indicate a virus.

REQUIREMENTS AND LIMITATIONS

To use SpamOracle, your mail must be delivered to a Unix machine on which you have a shell
account. This machine must have procmail(1) (see http://www.procmail.org/) installed.
Your ~/.forward file must be set up to run all incoming e-mail through procmail(1). If
your mail server supports the POP or IMAP protocols, you can also use fetchmail(1) to
fetch your mail from the server and have it delivered to your local machine.

To provide the corpus of messages from which SpamOracle "learns", an archive of about 1000
of your e-mails is needed. The archive must be manually or semi-automatically split into
known spams and known good messages. Mis-classified messages in the corpus (e.g. spams
mistakenly stored among the good messages) will decrease the efficiency of the
classification. The archive must be in Unix mailbox format, or in "one message per file"
format (a la MH). Other formats, such as Emacs' Babyl, are not supported.

The notion of "word" used by SpamOracle is slanted towards Western European languages,
i.e. the ISO Latin-1 and Latin-9 character sets. Preliminary support for JIS-encoded
Japanese can be selected at compile-time. SpamOracle will not work well if you receive
many legitimate e-mails written in other character sets, such as Chinese or Korean sets.

INITIALIZATION

To build the database of word frequencies from the corpus, do:

rm ~/.spamoracle.db
spamoracle add -v -good goodmails -spam spammails

By default, the database is stored in the file .spamoracle.db in your home directory.
This can be overriden with the -f option: spamoracle -f mydatabase add ... The -v option
prints progress information during the processing of the corpus.

This assumes that the good, non-spam messages from the corpus are stored in the file
goodmails, and the known spam messages in the file spammails. You can also fetch corpus
messages from several files, and/or process them via several invocations of SpamOracle:

spamoracle add -good goodmails1 ... goodmailsN
spamoracle add -spam spammails1 ... spammailsP

TESTING THE DATABASE

To check that the database was built correctly, and familiarize yourself with the
statistical analysis performed by SpamOracle, invoke the "test" mode on the mailboxes that
you just used for building the corpus:

spamoracle test goodmails | more
spamoracle test spammails | more

For each message in the given mailboxes, you'll see a summary like this:

From: bbo <[email protected]>
Subject: Check This Out
Score: 1.00 -- 15
Details: refid:98 $$$$:98 surfing:98 asp:95 click:93 cable:92
instantly:90 https:88 internet:87 www:86 U4:85 isn't:14 month:81
com:75 surf:75
Attachments: cset="GB2312" type="application/octet-stream"
name="Guangwen4.zip"
File: inbox/314

The first two lines are just the From: and Subject: fields of the original message.

The Score: line summarizes the result of the analysis. The first number (between 0.0 and
1.0) is the probability that the message is actually spam --- or, equivalently, the degree
of similarity of the message with the spam messages in the corpus. The second number (an
integer between 0 and 15) is the number of "interesting" words found in the message.
"Interesting" words are those that occur at least 5 times in the corpus. In the example,
we have 15 interesting words (the maximum) and a score of 1.00, indicating a spam with
high certainty.

The Details: line provides an explanation of the score. It lists the 15 most interesting
words found in the message, that is, the 15 interesting words whose probability of
denoting a spam is farthest away from the neutral 0.5. Each word is given with its
individual score, written as a percentage (between 01 and 99) rather than as a probability
so as to save space. Here, we see a number of very "spammish" words such as $$$$ or
click, with probability 0.98 and 0.93 respectively, and a few "innocent" words such as
isn't (probability 0.14). The U4 word with probability 0.85 is actually a pseudo-word
representing a 4-letter word all in uppercase -- something spammers are fond of.

The Attachments: line summarizes some information about MIME attachments for this message.
Here, we have one attachment of type application/octect-stream, file name Guangwen4.zip,
and character set GB2312 (an encoding for Chinese).

The File: line shows the file that is being tested.

Normally, when running spamoracle test goodmails, most messages should come out with low
score (0.2 or less), and when running spamoracle test spammails, most messages should come
out with a high score (0.8 or more). If not, your corpus isn't very good, or not well
classified into spam and non-spam. To quickly see the outliers, you can reduce the
interval of scores for which message summaries are displayed, as follows:

spamoracle test -min 0.2 goodmails | more
# Shows only good mails with score >= 0.2
spamoracle test -max 0.8 spammails | more
# Shows only spam mails with score <= 0.8

Now, for a more challenging test, take a mailbox that contains unfiltered e-mails, i.e. a
mixture of spam and legitimate e-mails, and run it through SpamOracle:

spamoracle test mymailbox | less

Marvel at how well the oracle recognizes spam from the rest! If the result isn't that
marvelous to you, keep in mind that certain spams are just too short to be recognized (not
enough significant words). Also, perhaps your corpus was too small, or not well
categorized...

MARKING AND FILTERING INCOMING E-MAIL

Once the database is built, you're ready to run incoming e-mails through SpamOracle. The
command spamoracle mark reads one e-mail from standard input, and copies it to standard
output, with two headers inserted: X-Spam: and X-Attachments:. The X-Spam: header has one
the following formats:

X-Spam: yes; score; details

or

X-Spam: no; score; details

or

X-Spam: unknown; score; details

The score and details are as described for spamoracle test.

The yes/no/unknown tag synthesizes the results of the analysis: yes means that the score
is >= 0.8 and at least 5 interesting words were found; no means that the score is <= 0.2
and at least 5 interesting words were found; unknown is returned otherwise. The unknown
case generally occurs for very short messages, where not enough interesting words were
found.

The X-Attachments: header contains the same information as the Attachments: output of
spamoracle test, that is, a summary of the message attachments.

To process automatically your incoming e-mail through SpamOracle and act upon the results
of the analysis, just insert the following "recipes" in the file ~/.procmailrc:

:0fw
| /usr/local/bin/spamoracle mark

:0
* ^X-Spam: yes;
spambox

What these cryptic commands mean is:

- Run every mail through the spamoracle mark command. (If spamoracle wasn't installed in
/usr/local/bin, adjust the path as necessary.) This adds two headers to the message: X-
Spam: and X-Attachments:, describing the results of the spam analysis and the attachment
analysis.

- If we have an X-Spam: yes header, deliver the message to the file spambox rather than to
your regular mailbox. Presumably, you'll read spambox once in a while, but less often
than your regular mailbox. Daring users can put /dev/null instead of spambox to just
throw away the message, but please don't do that until you've used SpamOracle for a while
and are happy with the results. SpamOracle's false positive rate (i.e. legitimate mails
classified as spam) is low (0.1%) but not null. So, better save the presumed spams
somewhere, and scan them quickly from time to time.

If you'd like to enjoy a bit of attachment-based filtering, here are some procmail rules
for that:

:0
* ^X-Attachments:.*name=".*\.(pif|scr|exe|bat|com)"
spambox

:0
* ^X-Attachments:.*type="audio/(x-wav|x-midi)
spambox

:0
* ^(Content-type:.*|X-Attachments:.*cset="|^Subject:.*=\?)(ks_c|gb2312|iso-2|euc-|big5|windows-1251)
spambox

The first rule treats as spam every mail that has a Windows executable as attachment.
These mails are typically sent by viruses. The second rule does the same with attachments
of type x-wav or x-midi. I never normally receive music by e-mail, however some popular
e-mail viruses seem fond of these attachment types. The third rule treats as spam every
mail that uses character encodings corresponding to Korean, Chinese, Japanese, and
Cyrillic.

UPDATING THE DATABASE

At any time, you can add more known spams or known legitimate messages to the database by
using the spamoracle add command.

For instance, if you find a spam message that was not classified as such, run it through
spamoracle add -spam, so that SpamOracle can learn from its mistake. (Without additional
arguments, this command will read a single message from standard input and record it as
spam.) Under mutt(1) for instance, just highlight the spam message and type

|spamoracle add -spam

Similarly, if you find a legitimate message while checking your spam box, run it through
spamoracle add -good.

Another option is to collect more known spams or more known good messages into mailbox
files, and once in a while do spamoracle add -good new_good_mails or spamoracle add -spam
new_spam_mails.

QUERYING THE DATABASE

For your edification and entertainment, the contents of the database can be queried by
regular expressions. The spamoracle list regexp command lists all words in the database
that match regexp (an Emacs-style regular expression), along with their number of
occurrences in spam mail and in good mail. For instance:

spamoracle list '.*' # show all words -- big list!
spamoracle list 'sex.*'
spamoracle list 'linux.*'

DATABASE BACKUPS

The database used by SpamOracle is stored in a compact, binary format that is not humanly
readable. Moreover, this format is subject to change in later versions of SpamOracle. To
facilitate backups and upgrades, the database contents can also be manipulated in a
portable, text format.

The spamoracle backup command dumps the contents of the database to standard output, in a
textual, portable format.

The spamoracle restore command reads such a dump from standard input and rebuilds the
database with this data.

The recommended procedure for upgrading to a newer version of SpamOracle is:

# Before the upgrade:
spamoracle backup > backupfile
# Upgrade SpamOracle
# Restore the database
spamoracle restore < backupfile

CONFIGURING FILTERING PARAMETERS

Many of the parameters that govern message classification can be configured via a
configuration file. By default, the configuration is read from the file .spamoracle.conf
in the user's home directory. A different configuration file can be specified on the
command line using the -config option: spamoracle -config myconfigfile ...

The list of configurable parameters and the format of the configuration file are described
in spamoracle.conf(5).

All parameters have reasonable defaults, but you can try to improve the quality of
classification further by tweaking them. To determine the impact of your changes, use
either the test or stat commands to spamoracle. The spamoracle stat command prints a one-
line summary of how many spam, non-spam, and unknown messages were found in the mailboxes
given as arguments.

TECHNICAL DETAILS

SpamOracle's notion of "word" is any run of 3 to 12 of the following characters: letters,
single quotes, and dashes (-). If support for non-English european languages was compiled
in, word characters also include the relevant accented letters for the languages in
question. All words are mapped to lowercase, and accented letters are mapped to the
corresponding non-accented letters.

A run of 3 to 12 of the following characters also constitutes a word: digits, dots,
commas, and dollar, Euro and percent signs.

In addition, a run of three or more uppercase letters generates a pseudo-word Un where n
is the length of the run. Similarly, a run of three or more non-ASCII characters (code >=
128) generates a pseudo-word Wn where n is the length of the run.

For instance, the following text:

SUMMER in English is written "ete" in French

is processed into the following words, assuming French support was selected at compile-
time:

U5 summer english written ete french W3

and if French support was not selected:

U5 summer english written french W3

To see the words that are extracted from a message, issue the spamoracle words command.
It reads either a single message from standard input, or all messages from the mailbox
files given as arguments, decomposes the messages into words and prints the words.

RANDOM NOTES

The database file can be compressed with gzip(1) to save disk space, at the expense of
slower spamoracle operations. If the database file specified with the -f option has the
extension .gz, spamoracle will automatically uncompress it on start-up, and re-compress it
after updates.

If your mail is stored in MH format, you may run into "command line too long" errors while
trying to process a lot of small files with the spamoracle add command, e.g. when doing
spamoracle add -good archives/*/* -spam spam/*
Instead, do something like:
find archives -type f -print | xargs spamoracle add -good
find spam -type f -print | xargs spamoracle add -spam

Use spamoracle online using onworks.net services