t-prot - Online in the Cloud

PROGRAM:

NAME

t-prot - TOFU Protection - Display Filter for RFC 5322 messages

SYNOPSIS

t-prot [OPTIONS]...

DESCRIPTION

This program is a filter to improve the readability of internet messages (emails and
usenet posts) by *hiding* some annoying parts, e.g. mailing list footers, signatures, and
TOFU (see definition below), as well as squeezing sequences of blank lines or punctuation.
The program also detects TOFU or a high quoting ratio in a message (so you may take
appropriate action, e.g. when submitting messages to a mailing list or a news server).
The filter is written in Perl and relies on input to be a single message conforming to RFC
822 or its successors, RFC 2822 and RFC 5322. In messages conforming to MIME (RFCs
2045-2049) t-prot handles text/plain parts, others are not touched.

Already reformatted messages are handled well: the script was initially designed to cope
with the output of the MUA mutt (which is the reason for not using standard CPAN modules
for handling messages).

T-prot offers example configuration files for mutt and its fork mutt-kz, Heirloom mailx
and metamail. Also coming with the t-prot package is the example S-Lang macro t-prot.sl
for using t-prot from within slrn. There is a proof-of-concept filter for INN2, which you
will have to adapt to the needs of the news site you host. For use with sendmail's
alias(5) file, please see below (the option -p provides an example line).

OPTIONS

If you do not specify any options, t-prot does ... nothing. Every feature you want must be
turned on explicitly. Admittedly, we have quite a number of options for t-prot. To limit
confusion they are grouped into five sections: Input/Output Options, Advertisement And
Mailing List Footers, Filtering Options, Detection Options, and Other Options. While the
others should be quite clear, filtering and detection might deserve a word (or two).

If you want to tune the appearance of your mail from within your MUA (or news messages
from within your NUA), then go for the Filtering Options section.

If you want to use t-prot to check on mails before they are submitted to mailing lists,
fed to your news server, or delivered by your MDA, then have a peek at the Detection
Options section. You may accept or reject/bounce messages depending on t-prot's result.

INPUT/OUTPUT OPTIONS

-i FILE
Defines an input file; default is '-' i.e. STDIN.

-o FILE
Defines the output file; default is STDOUT.

--body Input consists just of the message's body. There are no RFC 5322 header lines.

NOTE: This does not work with --pgp-short, and multipart messages will not be
detected due to missing headers.

--lax-security
Allow insecure writing method. DO NOT USE UNLESS YOU REALLY KNOW WHAT YOU ARE
DOING. (This ugly workaround is needed for some early mutt versions and should
NEVER be used as a default, otherwise it will probably turn into a security issue.)

You can use this option safely to enable -o /dev/null (or other files which cannot
be changed with the user's privileges).

--max-lines=x
Maximum number of lines a message may count (with headers). If the message is
longer than x lines, the message will not be processed but printed unmodified. Exit
status will be EX_DATAERR except when called with -Mmutt or -Mmutt-kz.

ADVERTISEMENT AND MAILING LIST FOOTERS

-a "commercial signature": Hides "footers" (signatures) from commercial email
providers.
This option compares the last lines of the message body with any footer file found
in the directory specified with -A DIRECTORY (which is mandatory for this option).
The comparison is done by perl's index() function (please try perldoc -f index for
details).

NOTE: This option is not needed if --ftr-ad is specified.

--ftr-ad
"enable aggressive ad footer matching": With this option enabled, t-prot makes
footer detection really greedy: We assume that commercial email providers aren't
even frightened to append changing texts *under* their ads which are appended to
the message body. Because these texts even have changing lengths we simply detect
the lines of the footer *anywhere* in the body of the message and assume that
everything below belongs to the footer. (Man, if life where always that easy! ;)
With this option even GMX ads should be easy to hide -- you buy this with a slight
performance hit (which is the reason this option is disabled by default), and with
the possibility that sometimes the algorithm is just a little *too* greedy.

NOTE: This requires a directory with footer files to be given with option -A
DIRECTORY.

-A DIRECTORY
"ad footer directory": Defines the directory which contains the advertisement list
footers (one footer per file) which are to be tested when removing them with
options -a or --ftr-ad.
This option is also needed if you do not want signature lengths to be counted wrong
or fullquotes get undetected when an ad footer is appended at the bottom of the
message (especially when using -S or -t).

-l "list signature": Hides "footers" (signatures) from mailing lists. Footer detection
works like the -a option.

NOTE: This requires a directory with footer files to be given with option -L
DIRECTORY. -l is not needed if --ftr-ml is specified.

--ftr-ml
"enable aggressive mailing list footer matching": With this option enabled t-prot
makes footer detection really greedy: Should be helpful with broken list servers,
or even if your email provider munges the bodies of your messages.
Works similar to --ftr-ad, just that it is intended for mailing list footers.

NOTE: This requires a directory with footer files to be given with option -L
DIRECTORY.

-L DIRECTORY
"list footer directory": Defines the directory which contains the mailing list
footers (one footer per file) which are to be tested when removing them with the
options -l or --ftr-ml.
This option is also needed if you do not want signature lengths to be counted wrong
or fullquotes get undetected when a mailing list footer is appended at the bottom
of the message (especially when using -S or -t).

FILTER OPTIONS

--bigq[=n[,x]]
"shrink big quotes": Blocks of quotes with more than n lines will be shrunk to x
lines. Defaults are 30 for n and 10 for x.

-c[n] "compress": Squeezes a sequence of blank lines to just n blank lines. n defaults to
2.

--diff Tolerate unified diff (see diff(1) and patch(1)) appended after the signature
(which usually makes the signature too long to be valid).

Also, protect diff standard output from hiding (which would otherwise be easy prey
for -t).

-e "ellipsis": Squeezes a sequence of four or more dots, exclamation marks, or
question marks to only three dots or marks, respectively.

--fixind
Fix broken quotes to adhere to RFC 3676 by removing spaces between quote characters
and adding a space after them.
NOTE: This may produce false positives if spaces in between quote characters are
intended (thus changing the quoting level, see RFC 3676 for details).

--groupwise
Hides TOFU as produced by Novell Groupwise.

-k "anti Kammquote": Tries (not too aggressively) to fix those broken zig-zag-shaped
lines wrapped around by some MUAs which are known as "Kammquoting" in German.

NOTE: This option is considered stable by now. However, sometimes Kammquotes should
have been removed but weren't. Please send a bug report if this happens to you
(after carefully reading the BUGS and REPORTING BUGS section of this man page, that
is).

Please also note that enabling this option is quite a performance hit.

--kdiff=n
Minimum length difference between two lines for wrapped line detection on
Kammquotes. For details, please see the source code.
Anyway, lower values make the algorithm more aggressive, higher values make
Kammquotes harder to detect. Default is 20.

Requires -k.

--kmaxl=n
Maximum line length for wrapped line detection on Kammquotes. For details, please
see the source code.
Anyway, higher values make the algorithm more aggressive, lower values make
Kammquotes harder to detect. Default is 80.

Requires -k.

--kminl=n
Minimum line length for wrapped line detection on Kammquotes. For details, please
see the source code.
Anyway, lower values make the algorithm more aggressive, higher values make
Kammquotes harder to detect. Default is 65.

Requires -k.

--locale=LOCALE
Specify which locale to use for correct parsing of your MUA's formatting of the
displayed message (usually it is the locale your MUA uses). Right now this option
is only used when -Mmutt or -Mmutt-kz is specified, but this may change in future.
You need the Perl module Locale::gettext for this feature.

NOTE: If you use mutt, mutt-kz or gnupg with locales, t-prot will only work
correctly if you specify the corresponding locale string. Alternatively, you can
use the environment variables LC_ALL, LC_MESSAGES, or LANG to specify the locale
string.

NOTE also: You also have to make sure you are running t-prot with matching gnupg
and mutt / mutt-kz versions. T-prot detects gnupg and mutt / mutt-kz locales of the
recent stable versions of those programs, earlier versions might not work well with
a recent version of t-prot.

-M, --muaMUA
"mail user agent": Turn on special treatment for some mail user agents. (Right now
only mutt(1) and mutt-kz(1) are supported, but more might be added in future.)
Caveat: If your MUA is supported by this feature you must ensure t-prot makes use
of it when called from within your MUA to work as desired.

-m "Microsoft TOFU": Hides TOFU as given by some Microsoft mailers. (You all surely
know these fullquotes beginning with
"----- Original Message -----"
and some header lines...)

--ms-smart
Burn CPU cycles trying to be smart with MS style TOFU: If there are PGP signed
parts inside the TOFU, the text still might conceal other message parts and
therefore should not be deleted.

Please note that this is probably just a waste of time because most MS Outlook
users who do produce this kind of TOFU won't care about making their messages the
least bit readable or even predictable. So this option will probably just be
interesting for mutt message hooks (to activate it on demand when you know the
sender tries to write legible messages).

Requires -Mmutt / -Mmutt-kz and -m.

--pgp-move
Move PGP and SSL verification output to bottom; requires -Mmutt / -Mmutt-kz.

--pgp-move-vrf
Move PGP and SSL verification output to bottom only if verification shows a good
signature and the signature could be verified as authentic (using a trust path). If
there is any problem with the signature, the PGP output should not be moved so the
user is more likely to notice. Requires -Mmutt / -Mmutt-kz.

NOTE: If gpg is terminated before finished (e.g. hitting Ctrl-C, or using kill(1)),
we cannot always detect if the check was interrupted. Though t-prot tries to be
smart, there will be false positives.

--pgp-short
Hide non-relevant PGP key uids; requires -Mmutt / -Mmutt-kz.

-r "rip header off": Hides all mail header lines.

--reply
Subject lines with multiple reply prefixes (Re: and translations into other
languages) get squeezed to only one prefix.

-S[n] "suppression of overlong signatures": Signatures are to be n lines (not including
the one containing dash-dash-space) or less. If there are more, it is probably not
that spirited after all. So with this option you trade it for a truely nice line.
If no n is given, default is 4. (We do not recommend using a value other than 4.
Consider this old-fashioned, but we actually do *like* RFC conformance.)

NOTE: The line containing "-- " ist not counted when testing for an overlong
signature, but it is included when displaying how many lines were deleted.

-s "signature deletion": Hides signatures, i.e. all lines after a "signature dashes"
line, i.e. a line with three characters: dash-dash-space (no more, no less).

--sani Sanitize headers "To:", "From:" and "Subject:": Quoted-printable gets fixed to the
corresponding chars. German Umlauts are translated to their "ae", "oe", "ue"
pendants.
Useful e.g. for searching by subject within MUAs like Berkeley mailx.

--sigsmax[=n]
"maximum number of tolerated signatures": Here you can define how many signatures
you accept to be treated as such. (Most significant behaviour is when microsoft
style quotes are removed. Experts please see the code for the more subtle
implications of this option.)
Leave empty or specify zero to have an unlimited number of sigs. Default is 1.

--spass
"SpamAssassin workaround": SpamAssassin (available at http://spamassassin.org/)
often is configured that it adds some lines to the message body containing
information about the spam criteria which were found matching for the message. This
option enables an extra test to avoid false positives for Microsoft style TOFU on
such messages.

-t "TOFU deletion": Hides "traditional style" TOFU, where each line begins with the
indent string ">".

-w "whitespace deletion": Hides trailing whitespace (sequences of space and tab).
CAVEAT: This may lead to interesting effects with crossposts between mailing lists
or with undetected signature attempts.

DETECTION OPTIONS

-P MESSAGE
"user defined bounce message for picky delivery": You may specify your own bounce
message to be returned when we try to deliver an email and bounce it because there
is TOFU inside. See -p.

-p [ADDRESS]
"picky delivery": If we really find some TOFU, abort with exit code EX_UNAVAILABLE.
Otherwise redirect the message to ADDRESS if given.

Intended for use from within mail delivery agents (MDAs) or mail transport agents
(MTAs), or even from within INN, so the message bounces if TOFU is detected, and
does not get on *your* nerves. :)

As an example for usage with sendmail, put this line into your alias file and
invoke newaliases:

notofu: |"/usr/local/bin/t-prot -mt -p=user@mydomain"

This will bounce messages for <notofu@domainname> if any TOFU is detected inside
the message, and deliver it to <user@mydomain> otherwise. Note that TOFU is only
detected if you specify -t respectively -m.

PLEASE be careful not to bounce messages to mailing lists!

--check[=FLAGS]
Run checks. If successful, print an error message and quit with an appropriate exit
code. Useful e.g. for rejecting messages from within INN2.

Flags are separated by commas (no whitespaces), and can be the following (right now
just one flag):

ratio[=n]
If the quoting ratio is n or more, the message is rejected. Must be between 0 and
1, or else it is entirely disabled. Default is 0.75 (i.e., 75% of the message lines
are quotes).

-d, --debug
Print envelope info to syslog when bouncing TOFU contaminated email. Default
syslog facility is mail.debug. Requires -p.

OTHER OPTIONS

-h, --help
Displays a short help text with a summary on all options, and exits.

-v, --version
Prints the current version number and exits.

ENVIRONMENT

The environment variables LC_ALL, LC_MESSAGES, and LANG are read and respected when
interpreting output by mutt / mutt-kz or gnupg (unless they are overruled by the --locale
option). T-prot's own output is English regardless of any locale setting.

EXIT STATUS

On program exit, t-prot uses exit codes from /usr/include/sysexits.h and thus behaves in a
manner that sendmail and others understand when calling t-prot.

Currently, the codes used are
EX_OK
EX_USAGE
EX_DATAERR
EX_UNAVAILABLE
EX_SOFTWARE
EX_IOERR

If, however, perl fails to compile and execute t-prot, perl's normal exit codes will be
returned.

TOFU?

TOFU is an abbreviation which mixes German and English words; it expands to "text oben,
full-quote unten" which means "text above - full quote below" and describes the style of
so many users who let their mailer or newsreader quote everything of the previous message
and just add some text at the top; obviously they think that quoted text must not be
changed at all. This is quite annoying as it needlessly sends a lot of data even when it
is not required. Some editing of messages is desired. Please point these people to the
page http://www.river.com/users/share/etiquette/edit.html - thank you!

PERFORMANCE

There are several ways to fine-tune t-prot's performance:

Some command line options are quite grave a performance hit -- do not use -k and
especially --ms-smart if you are content without them.

Checking for special footers is very costly as well. Put as few footer files as absolutely
needed in any footer directory.

All PGP related options are eating up lots of CPU time. Try to avoid them on unsigned and
unencrypted messages.

When calling t-prot from within mutt (or mutt-kz), you might use mutt's folder-hook and
message-hook facilities to turn options on only when needed, e.g. to set up a different
footer directory for each mailing list folder.

TROUBLESHOOTING

Q: I want to make my mailing list footer files match more different mailing list
footers. Can I use regular expressions, or how can I accomplish that?

A: Nope, regexp's do not work here. The comparison is made by the perl builtin index()
function (see perldoc for more detailed info), so you must exactly match the
beginning of the line. The longer the line you specify, the more precise the match;
if your line is empty you match unconditionally.

Q: I use the options -l and -L to suppress mailing list footers when displaying
messages in mutt(1). This does work sometimes, but sometimes it does not: the
footer is not detected, and therefore full quotes are not deleted and signatures
are detected as too long (which they aren't).

A: This might occur if the message is badly encoded, so mutt cannot resolve all
encoded characters, e.g. if you have an encoded message on a mailing list, and
majordomo appends a mailing list footer in a different encoding (or even plain us-
ascii). "-- " simply does not match "--=20".
Another problem are non-us-ascii characters. Just avoid them, and everything should
work fine.
See the preceding Q+A for a solution.

Q: I want to write a message which contains parts that should *not* be deleted even
when filtered with t-prot. Is this possible?

A: Yes, but please do not spread word of it. Make unobstrusive use of the verbatim
instruction:

#v+
This line is protected from being filtered by t-prot !!!!!!!
#v-
Text coming now is not.

Use t-prot online using onworks.net services