mailtoe - Online in the Cloud

PROGRAM:

NAME

mailtoe - a train-on-error simulator for use with dbacl.

SYNOPSIS

mailtoe command [ command_arguments ]

DESCRIPTION

mailtoe automates the task of testing email filtering and classification programs such as
dbacl(1). Given a set of categorized documents, mailtoe initiates test runs to estimate
the classification errors and thereby permit fine tuning of the parameters of the
classifier.

Train-on-error (TOE) is a learning method which is sometimes advocated for email
classifiers. Given an incoming email stream, the method consists in reusing a fixed set of
category databases until the first misclassification occurs. At that point, the offending
email is used to relearn the relevant category, until the next misclassification. In this
way, categories are only updated when errors occur. This directly models the way that some
email classifiers are used in practice.

TOE's error rates depend directly on the order in which emails are seen. A small change
in ordering, as might happen due to networking delays, can have a large impact on the
number of misclassifications. Consequently, mailtoe does not give meaningful results,
unless the sample emails are chosen carefully. However, as this method is commonly used
by spam filters, it is still worth computing to foster comparisons. Other methods (see
mailcross(1),mailfoot(1)) attempt to capture the behaviour of classification errors in
other ways.

To improve and stabilize the error rate calculation, mailtoe performs the TOE simulations
several times on slightly reordered email streams, and averages the results. The
reorderings occur by multiplexing the emails from each category mailbox in random order.
Thus if there are three categories, the first email classified is chosen randomly from the
front of the sample email streams of each type. The second email is also chosen randomly
among the three types, from the front of the
streams after the first email was removed. Simulation stops when all sample streams are
exhausted.

mailtoe uses the environment variable MAILTOE_FILTER when executing, which permits the
simulation of arbitrary filters, provided these satisfy the compatibility conditions
stated in the ENVIRONMENT section below.

For convenience, mailtoe implements a testsuite framework with predefined wrappers for
several open source classifiers. This permits the direct comparison of dbacl(1) with
competing classifiers on the same set of email samples. See the USAGE section below.

During preparation, mailtoe builds a subdirectory named mailtoe.d in the current working
directory. All needed calculations are performed inside this subdirectory.

EXIT STATUS

mailtoe returns 0 on success, 1 if a problem occurred.

COMMANDS

prepare size
Prepares a subdirectory named mailtoe.d in the current working directory, and
populates it with empty subdirectories for exactly size subsets.

add category [ FILE ]...
Takes a set of emails from either FILE if specified, or STDIN, and associates them
with category. The ordering of emails within FILE is preserved, and subsequent
FILEs are appended to the first in each category. This command can be repeated
several times, but should be executed at least once.

clean Deletes the directory mailtoe.d and all its contents.

run Multiplexes randomly from the email streams added earlier, and relearns categories
only when a misclassification occurs. The simulation is repeated size times.

summarize
Prints average error rates for the simulations.

plot [ ps | logscale ]...
Plots the number of errors over simulation time. The "ps" option, if present,
writes the plot to a postscript file in the directory mailtoe/plots, instead of
being shown on-screen. The "logscale" option, if present, causes the plot to be on
the log scale for both ordinates.

review truecat predcat
Scans the last run statistics and extracts all the messages which belong to
category truecat but have been classified into category predcat. The extracted
messages are copied to the directory mailtoe.d/review for perusal.

testsuite list
Shows a list of available filters/wrapper scripts which can be selected.

testsuite select [ FILTER ]...
Prepares the filter(s) named FILTER to be used for simulation. The filter name is
the name of a wrapper script located in the directory /usr/share/dbacl/testsuite.
Each filter has a rigid interface documented below, and the act of selecting it
copies it to the mailtoe.d/filters directory. Only filters located there are used
in the simulations.

testsuite deselect [ FILTER ]...
Removes the named filter(s) from the directory mailtoe.d/filters so that they are
not used in the simulation.

testsuite run [ plots ]
Invokes every selected filter on the datasets added previously, and calculates
misclassification rates. If the "plots" option is present, each filter simulation
is plotted as a postscript file in the directory mailtoe.d/plots.

testsuite status
Describes the scheduled simulations.

testsuite summarize
Shows the cross validation results for all filters. Only makes sense after the run
command.

USAGE

The normal usage pattern is the following: first, you should separate your email
collection into several categories (manually or otherwise). Each category should be
associated with one or more folders, but each folder should not contain more than one
category. Next, you should decide how many runs to use, say 10. The more runs you use,
the better the predicted error rates. However, more runs take more time. Now you can type

% mailtoe prepare 10

Next, for every category, you must add every folder associated with this category. Suppose
you have three categories named spam, work, and play, which are associated with the mbox
files spam.mbox, work.mbox, and play.mbox respectively. You would type

% mailtoe add spam spam.mbox
% mailtoe add work work.mbox
% mailtoe add play play.mbox

You should aim for a similar number of emails in each category, as the random multiplexing
will be unbalanced otherwise. The ordering of the email messages in each *.mbox file is
important, and is preserved during each simulation. If you repeatedly add to the same
category, the later mailboxes will be appended to the first, preserving the implied
ordering.

You can now perform as many TOE simulations as desired. The multiplexed emails are
classified and learned one at a time, by executing the command given in the environment
variable MAILTOE_FILTER. If not set, a default value is used.

% mailtoe run
% mailtoe summarize

The testsuite commands are designed to simplify the above steps and allow comparison of a
wide range of email classifiers, including but not limited to dbacl. Classifiers are
supported through wrapper scripts, which are located in the /usr/share/dbacl/testsuite
directory.

The first stage when using the testsuite is deciding which classifiers to compare. You
can view a list of available wrappers by typing:

% mailtoe testsuite list

Note that the wrapper scripts are NOT the actual email classifiers, which must be
installed separately by your system administrator or otherwise. Once this is done, you
can select one or more wrappers for the simulation by typing, for example:

% mailtoe testsuite select dbaclA ifile

If some of the selected classifiers cannot be found on the system, they are not selected.
Note also that some wrappers can have hard-coded category names, e.g. if the classifier
only supports binary classification. Heed the warning messages.

It remains only to run the simulation. Beware, this can take a long time (several hours
depending on the classifier).

% mailtoe testsuite run
% mailtoe testsuite summarize

Once you are all done, you can delete the working files, log files etc. by typing

% mailtoe clean

SCRIPT INTERFACE

mailtoe testsuite takes care of learning and classifying your prepared email corpora for
each selected classifier. Since classifiers have widely varying interfaces, this is only
possible by wrapping those interfaces individually into a standard form which can be used
by mailtoe testsuite.

Each wrapper script is a command line tool which accepts a single command followed by zero
or more optional arguments, in the standard form:

wrapper command [argument]...

Each wrapper script also makes use of STDIN and STDOUT in a well defined way. If no
behaviour is described, then no output or input should be used. The possible commands are
described below:

filter In this case, a single email is expected on STDIN, and a list of category filenames
is expected in $2, $3, etc. The script writes the category name corresponding to
the input email on STDOUT. No trailing newline is required or expected.

learn In this case, a standard mbox stream is expected on STDIN, while a suitable
category file name is expected in $2. No output is written to STDOUT.

clean In this case, a directory is expected in $2, which is examined for old database
information. If any old databases are found, they are purged or reset. No output is
written to STDOUT.

describe
IN this case, a single line of text is written to STDOUT, describing the filter's
functionality. The line should be kept short to prevent line wrapping on a
terminal.

bootstrap
In this case, a directory is expected in $2. The wrapper script first checks for
the existence of its associated classifier, and other prerequisites. If the check
is successful, then the wrapper is cloned into the supplied directory. A courtesy
notification should be given on STDOUT to express success or failure. It is also
permissible to give longer descriptions caveats.

toe In this case, a list of categories is expected in $3, $4, etc. Every possible
category must be listed. Preceding this list, the true category is given in $2.

foot Used by mailfoot(1).

ENVIRONMENT

Right after loading, mailtoe reads the hidden file .mailtoerc in the $HOME directory, if
it exists, so this would be a good place to define custom values for environment
variables.

MAILTOE_FILTER
This variable contains a shell command to be executed repeatedly during the running
stage. The command should accept an email message on STDIN and output a resulting
category name. On the command line, it should also accept first the true category
name, then a list of all possible category file names. If the output category does
not match the true category, then the relevant categories are assumed to have been
silently updated/relearned. If MAILTOE_FILTER is undefined, mailtoe uses a default
value.

TEMPDIR
This directory is exported for the benefit of wrapper scripts. Scripts which need
to create temporary files should place them a the location given in TEMPDIR.

NOTES

The subdirectory mailtoe.d can grow quite large. It contains a full copy of the training
corpora, as well as learning files for size times all the added categories, and various
log files.

While TOE simulations for dbacl(1) can be used to compare with other classifiers, TOE
should not be used for real world classifications. This is because, unlike many other
filters, dbacl(1) learns evidence weights in a nonlinear way, and does not preserve
relative weights between tokens, even if those tokens aren't seen in new emails.

WARNING

Because the ordering of emails within the added mailboxes matters, the estimated error
rates are not well defined or even meaningful in an objective sense. However, if the
sample emails represent an actual snapshot of a user's incoming email, then the error
rates are somewhat meaningful. The simulations can then be interpreted as alternate
realities where a given classifier would have intercepted the incoming mail.

SOURCE

The source code for the latest version of this program is available at the following
locations:

http://www.lbreyer.com/gpl.html
http://dbacl.sourceforge.net

Use mailtoe online using onworks.net services