amavis-logwatch - Online in the Cloud

PROGRAM:

NAME

amavis-logwatch - An Amavisd-new log parser and analysis utility

SYNOPSIS

amavis-logwatch [options] [logfile ...]

DESCRIPTION

The amavis-logwatch(1) utility is an Amavisd-new log parser that produces summaries,
details, and statistics regarding the operation of Amavisd-new (henceforth, simply called
Amavis).

This utility can be used as a standalone program, or as a Logwatch filter module to
produce Amavisd-new summary and detailed reports from within Logwatch.

Amavis-logwatch is able to produce a wide range of reports with data grouped and sorted as
much as possible to reduce noise and highlight patterns. Brief summary reports provide a
quick overview of general Amavis operations and message delivery, calling out warnings
that may require attention. Detailed reports provide easy to scan, hierarchically-
arranged and organized information, with as much or little detail as desired.

Much of the interesting data is available when Amavis' $log_level is set to at least 2.
See Amavis Log Level below.

Amavis-logwatch outputs two principal sections: a Summary section and a Detailed section.
For readability and quick scanning, all event or hit counts appear in the left column,
followed by brief description of the event type, and finally additional statistics or
count representations may appear in the rightmost column.

The following segment from a sample Summary report illustrates:

****** Summary ********************************************

9 Miscellaneous warnings

20313 Total messages scanned ---------------- 100.00%
1008.534M Total bytes scanned 1,057,524,252
======== ================================================

1190 Blocked ------------------------------- 5.86%
18 Malware blocked 0.09%
4 Banned name blocked 0.02%
416 Spam blocked 2.05%
752 Spam discarded (no quarantine) 3.70%

19123 Passed -------------------------------- 94.14%
47 Bad header passed 0.23%
19076 Clean passed 93.91%
======== ================================================

18 Malware ------------------------------- 0.09%
18 Malware blocked 0.09%

4 Banned -------------------------------- 0.02%
4 Banned file blocked 0.02%

1168 Spam ---------------------------------- 5.75%
416 Spam blocked 2.05%
752 Spam discarded (no quarantine) 3.70%

19123 Ham ----------------------------------- 94.14%
47 Bad header passed 0.23%
19076 Clean passed 93.91%
======== ================================================

1982 SpamAssassin bypassed
32 Released from quarantine
2 DSN notification (debug supplemental)
2 Bounce unverifiable
2369 Whitelisted
2 Blacklisted
12 MIME error
58 Bad header (debug supplemental)
40 Extra code modules loaded at runtime

The report indicates there were 9 general warnings, and Amavis scanned a total of 20313
messages for a total of 1008.53 megabytes or 1,057,524,252 bytes. The next summary groups
shows the Blocked / Passed overview, with 1190 Blocked messages (broken down as 18
messages blocked as malware, 4 messages with banned names, 416 spam messages, and 752
discarded messages), and 19123 Passed messages (47 messages with bad headers and 19076
clean messages).

The next (optional) summary grouping shows message disposition by contents category.
There were 18 malware messages and 4 banned file messages (all blocked), 1168 Spam
messages, of which 416 were blocked (quarantined) and 752 discarded. Finally, there were
19123 messages consdidered to be Ham (i.e. not spam), 47 of which contained bad headers.

Additional count summaries for a variety of events are also listed.

There are dozens of sub-sections available in the Detailed report, each of whose output
can be controlled in various ways. Each sub-section attempts to group and present the
most meaningful data at superior levels, while pushing less useful or noisy data towards
inferior levels. The goal is to provide as much benefit as possible from smart grouping
of data, to allow faster report scanning, pattern identification, and problem solving.
Data is always sorted in descending order by count, and then numerically by IP address or
alphabetically as appropriate.

The following Spam blocked segment from a sample Detailed report illustrates the basic
hierarchical level structure of amavis-logwatch:

****** Detailed *******************************************

19346 Spam blocked -----------------------------------
756 [email protected]
12 10.0.0.2
12 <>
12 192.168.2.2
12 <>
5 192.168.2.1
...

The amavis-logwatch utility reads from STDIN or from the named Amavis logfile. Multiple
logfile arguments may be specified, each processed in order. The user running amavis-
logwatch must have read permission on each named log file.

Options
The options listed below affect the operation of amavis-logwatch. Options specified later
on the command line override earlier ones. Any option may be abbreviated to an
unambiguous length.

--[no]autolearn
--show_autolearn boolean
Enables (disables) output of the autolearn report. This report is only available
if the default Amavis $log_templ has been modified to provide autolearn results in
log entries. This can be done by uncommenting two lines in the Amavis program
itself (where the default log templates reside), or by correctly adding the
$log_templ variable to the amavisd.conf file. See Amavis' README.customize and
search near the end of the Amavisd program for "autolearn".

--[no]by_ccat_summary
--show_by_ccat_summary boolean
Enables (disables) the by contents category summary in the Summary section.
Default: enabled.

-f config_file
--config_file config_file
Use an alternate configuration file config_file instead of the default. This
option may be used more than once. Multiple configuration files will be processed
in the order presented on the command line. See CONFIGURATION FILE below.

--debug keywords
Output debug information during the operation of amavis-logwatch. The parameter
keywords is one or more comma or space separated keywords. To obtain the list of
valid keywords, use --debug xxx where xxx is any invalid keyword.

--detail level
Sets the maximum detail level for amavis-logwatch to level. This option is global,
overriding any other output limiters described below.

The amavis-logwatch utility produces a Summary section, a Detailed section, and
additional report sections. With level less than 5, amavis-logwatch will produce
only the Summary section. At level 5 and above, the Detailed section, and any
additional report sections are candidates for output. Each incremental increase in
level generates one additional hierarchical sub-level of output in the Detailed
section of the report. At level 10, all levels are output. Lines that exceed the
maximum report width (specified with max_report_width) will be cut. Setting level
to 11 will prevent lines in the report from being cut (see also --line_style).

--[no]first_recip_only
--show_first_recip_only boolean
Specifies whether or not to sort by, and show, only the first recipient when a
scanned messages contains multiple recipients.

--help Print usage information and a brief description about command line options.

--ipaddr_width width
Specifies that IP addresses in address/hostname pairs should be printed with a
field width of width characters. Increasing the default may be useful for systems
using long IPv6 addresses.

-l limiter=levelspec
--limit limiter=levelspec
Sets the level limiter limiter with the specification levelspec.

--line_style style
Specifies how to handle long report lines. Three styles are available: full,
truncate, and wrap. Setting style to full will prevent cutting lines to
max_report_width; this is what occurs when detail is 11 or higher. When style is
truncate (the default), long lines will be truncated according to max_report_width.
Setting style to wrap will wrap lines longer than max_report_width such that left
column hit counts are not obscured. This option takes precedence over the line
style implied by the detail level. The options --full, --truncate, and --wrap are
synonyms.

--nodetail
Disables the Detailed section of the report, and all supplemental reports. This
option provides a convenient mechanism to quickly disable all sections under the
Detailed report, where subsequent command line options may re-enable one or more
sections to create specific reports.

--sarules `S,H´
--sarules default
Enables the SpamAssassin Rules Hit report. The comma-separated S and H arguments
are top N values for the Spam and Ham reports, respectively, and can be any integer
greater than or equal to 0, or the keyword all. The keyword default uses the
built-in default values.

--nosarules
Disables the SpamAssassin Rules Hit report.

--sa_timings nrows
Enables the SpamAssassin Timings percentiles report. The report can be limited to
the top N rows with the nrows argument. This report requires Amavis 2.6+ and
SpamAssassin 3.3+.

--sa_timings_percentiles `P1 [P2 ...]´
Specifies the percentiles shown in the SpamAssassin Timings report. The arguments
P1 ... are integers from 0 to 100 inclusive. Their order will be preserved in the
report.

--nosa_timings
Disables the SpamAssassin Timings report.

--version
Print amavis-logwatch version information.

--score_frequencies `B1 [B2 ...]´
--score_frequencies default
Enables the Spam Score Frequency report. The arguments B1 ... are frequency
distribution buckets, and can be any real numbers. Their order will be preserved
in the report. The keyword default uses the built-in default values.

--noscore_frequencies
Disables the Spam Score Frequency report.

--score_percentiles `P1 [P2 ...]´
--score_percentiles default
Enables the Spam Score Percentiles report. The arguments P1 ... specify the
percentiles shown in the report, and are integers from 0 to 100 inclusive. The
keyword default uses the built-in default values.

--noscore_percentiles
Disables the Spam Score Percentiles report.

--[no]sect_vars
--show_sect_vars boolean
Enables (disables) supplementing each Detailed section title with the name of that
section's level limiter. The name displayed is the command line option (or
configuration file variable) used to limit that section's output. With the large
number of level limiters available in amavis-logwatch, this a convenient mechanism
for determining exactly which level limiter affects a section.

--[no]startinfo
--show_startinfo boolean
Enables (disables) the Amavis startup report showing most recent Amavis startup
details.

--[no]summary

--show_summary
Enables (disables) displaying of the the Summary section of the report. The
variable Amavis_Show_Summary in used in a configuration file.

--syslog_name namepat
Specifies the syslog service name that amavis-logwatch uses to match syslog lines.
Only log lines whose service name matches the perl regular expression namepat will
be used by amavis-logwatch; all non-matching lines are silently ignored. This is
useful when a pre-installed Amavis package uses a name other than the default
(amavis).

Note: if you use parenthesis in your regular expression, be sure they are
cloistering and not capturing: use (?:pattern) instead of (pattern).

--timings percent
Enables the Amavis Scan Timings percentiles report. The report can be top N-
percent limited with the percent argument.

--timings_percentiles `P1 [P2 ...]´
Specifies the percentiles shown in the Scan Timings report. The arguments P1 ...
are integers from 0 to 100 inclusive. Their order will be preserved in the report.

--notimings
Disables the Amavis Scan Timings report.

--version
Print amavis-logwatch version information.

Level Limiters
The output of every section in the Detailed report is controlled by a level limiter. The
name of the level limiter variable will be output when the sect_vars option is set. Level
limiters are set either via command line in standalone mode with --limit limiter=levelspec
option, or via configuration file variable $amavis_limiter=levelspec. Each limiter
requires a levelspec argument, which is described below in LEVEL CONTROL.

The list of level limiters is shown below.

Amavis major contents category (ccatmajor) sections, listed in order of priority: VIRUS,
BANNED, UNCHECKED, SPAM, SPAMMY, BADH, OVERSIZED, MTA, CLEAN.

MalwareBlocked
MalwarePassed
Blocked or passed messages that contain malware (ccatmajor: VIRUS).

BannedNameBlocked
BannedNamePassed
Blocked or passed messages that contain banned names in MIME parts (ccatmajor:
BANNED).

UncheckedBlocked
UncheckedPassed
Blocked or passed messages that were not checked by a virus scanner or SpamAssassin
(Amavis ccatmajor: UNCHECKED).

SpamBlocked
SpamPassed
Blocked or passed messages that were considered spam that reached kill level
(Amavis ccatmajor: SPAM)

SpammyBlocked
SpammyPassed
Blocked or passed messages that were considered spam, but did not reach kill level
(Amavis ccatmajor: SPAMMY)

BadHeaderBlocked
BadHeaderPassed
Blocked or passed messages that contain bad mail headers (ccatmajor: BAD-HEADER).

OversizedBlocked
OversizedPassed
Blocked or passed messages that were considered oversized (Amavis ccatmajor:
OVERSIZED).

MtaBlocked
MtaPassed
Blocked or passed messages due to failure to re-inject to MTA (Amavis ccatmajor:
MTA-BLOCKED). Occurrences of this event indicates a configuration problem. [
note: I don't believe mtapassed occurs, but exists for completeness.]

OtherBlocked
OtherPassed
Blocked or passed messages that are not any of other major contents categories
(Amavis ccatmajor: OTHER).

TempFailBlocked
TempfailPassed
Blocked or passed messages that had a temporary failure (Amavis ccatmajor:
TEMPFAIL)

CleanBlocked
CleanPassed
Messages blocked or passed which were considered clean (Amavis ccatmajor: CLEAN;
i.e. non-spam, non-viral).

Other sections, arranged alphabetically:

AvConnectFailure
Problems connecting to Anti-Virus scanner(s).

AvTimeout
Timeouts awaiting responses from Anti-Virus scanner(s).

ArchiveExtract
Archive extraction problems.

BadHeaderSupp
Supplemental debug information regarding messages containing bad mail headers.

Bayes Messages frequencies by Bayesian probability buckets.

BadAddress
Invalid mail address syntax.

Blacklisted
Messages that were (soft-)blacklisted. See also Whitelisted below.

BounceKilled
BounceRescued
BounceUnverifiable
Disposition of incoming bounce messages (DSNs).

ContentType
MIME attachment breakdown by type/subtype.

DccError
Errors encountered with or returned by DCC.

DefangError
Errors encountered during defang process.

Defanged
Messages defanged (rendered harmless).

DsnNotification
Errors encountered during attempt to send delivery status notification.

DsnSuppressed
Delivery status notification (DSN) intentionally suppressed.

ExtraModules
Additional code modules Amavis loaded during runtime.

FakeSender
Forged sender addresses, as determimed by Amavis.

Fatal Fatal events. These are presented at the top of the report, as they may require
attention.

LocalDeliverySkipped
Failures delivering to a local address.

MalwareByScanner
Breakdown of malware by scanner(s) that detected the malware.

MimeError
Errors encountered during MIME extraction.

Panic Panic events. These are presented at the top of the report, as they may require
attention.

p0f Passive fingerprint (p0f) hits, grouped by mail contents type (virus, unchecked,
banned, spam, ham), next by operating system genre, and finally by IP address.
Note: Windows systems are refined by Windows OS version, whereas versions of other
operating systems are grouped generically.

Released
Messages that were released from Amavis quarantine.

SADiags
Diagnostics as reported from SpamAssassin.

SmtpResponse
SMTP responses received during dialog with MTA. These log entries are primarly
debug.

TmpPreserved
Temporary directories preserved by Amavis when some component encounters a problem
or failure. Directories listed and their corresponding log entries should be
evaluated for problems.

VirusScanSkipped
Messages that could not be scanned by a virus scanner.

Warning
Warning events not categorized in specific warnings below. These are presented at
the top of the report, as they may require attention.

WarningAddressModified
Incomplete email addresses modified by Amavis for safety.

WarningNoQuarantineId
Attempts to release a quarantined message that did not contain an X-Quarantine-ID
header.

WarningSecurity levelspec
Insecure configuration or utility used by Amavis.

WarningSmtpShutdown
Failures during SMTP conversation with MTA.

WarningSql
Failures to communicate with, or error replies from, SQL service.

Whitelisted
Messages that were (soft-)whitelisted. See also Blacklisted above.

LEVEL CONTROL

The Detailed section of the report consists of a number of sub-sections, each of which is
controlled both globally and independently. Two settings influence the output provided in
the Detailed report: a global detail level (specified with --detail) which has final (big
hammer) output-limiting control over the Detailed section, and sub-section specific detail
settings (small hammer), which allow further limiting of the output for a sub-section.
Each sub-section may be limited to a specific depth level, and each sub-level may be
limited with top N or threshold limits. The levelspec argument to each of the level
limiters listed above is used to accomplish this.

It is probably best to continue explanation of sub-level limiting with the following well-
known outline-style hierarchy, and some basic examples:

level 0
level 1
level 2
level 3
level 4
level 4
level 2
level 3
level 4
level 4
level 4
level 3
level 4
level 3
level 1
level 2
level 3
level 4

The simplest form of output limiting suppresses all output below a specified level. For
example, a levelspec set to "2" shows only data in levels 0 through 2. Think of this as
collapsing each sub-level 2 item, thus hiding all inferior levels (3, 4, ...), to yield:

level 0
level 1
level 2
level 2
level 1
level 2

Sometimes the volume of output in a section is too great, and it is useful to suppress any
data that does not exceed a certain threshold value. Consider a dictionary spam attack,
which produces very lengthy lists of hit-once recipient email or IP addresses. Each sub-
level in the hierarchy can be threshold-limited by setting the levelspec appropriately.
Setting levelspec to the value "2::5" will suppress any data at level 2 that does not
exceed a hit count of 5.

Perhaps producing a top N list, such as top 10 senders, is desired. A levelspec of
"3:10:" limits level 3 data to only the top 10 hits.

With those simple examples out of the way, a levelspec is defined as a whitespace- or
comma-separated list of one or more of the following:

l Specifies the maximum level to be output for this sub-section, with a range from 0
to 10. if l is 0, no levels will be output, effectively disabling the sub-section
(level 0 data is already provided in the Summary report, so level 1 is considered
the first useful level in the Detailed report). Higher values will produce output
up to and including the specified level.

l.n Same as above, with the addition that n limits this section's level 1 output to the
top n items. The value for n can be any integer greater than 1. (This form of
limiting has less utility than the syntax shown below. It is provided for backwards
compatibility; users are encouraged to use the syntax below).

l:n:t This triplet specifies level l, top n, and minimum threshold t. Each of the values
are integers, with l being the level limiter as described above, n being a top n
limiter for the level l, and t being the threshold limiter for level l. When both
n and t are specified, n has priority, allowing top n lists (regardless of
threshold value). If the value of l is omitted, the specified values for n and/or
t are used for all levels available in the sub-section. This permits a simple form
of wildcarding (eg. place minimum threshold limits on all levels). However,
specific limiters always override wildcard limiters. The first form of level
limiter may be included in levelspec to restrict output, regardless of how many
triplets are present.

All three forms of limiters are effective only when amavis-logwatch's detail level is 5 or
greater (the Detailed section is not activated until detail is at least 5).

See the EXAMPLES section for usage scenarios.

CONFIGURATION FILE

Amavis-logwatch can read configuration settings from a configuration file. Essentially,
any command line option can be placed into a configuration file, and these settings are
read upon startup.

Because amavis-logwatch can run either standalone or within Logwatch, to minimize
confusion, amavis-logwatch inherits Logwatch's configuration file syntax requirements and
conventions. These are:

· White space lines are ignored.

· Lines beginning with # are ignored

· Settings are of the form:

option = value

· Spaces or tabs on either side of the = character are ignored.

· Any value protected in double quotes will be case-preserved.

· All other content is reduced to lowercase (non-preserving, case insensitive).

· All amavis-logwatch configuration settings must be prefixed with "$amavis_" or amavis-
logwatch will ignore them.

· When running under Logwatch, any values not prefixed with "$amavis_" are consumed by
Logwatch; it only passes to amavis-logwatch (via environment variable) settings it
considers valid.

· The values True and Yes are converted to 1, and False and No are converted to 0.

· Order of settings is not preserved within a configuration file (since settings are
passed by Logwatch via environment variables, which have no defined order).

To include a command line option in a configuration file, prefix the command line option
name with the word "$amavis_". The following configuration file setting and command line
option are equivalent:

$amavis_Line_Style = Truncate

--line_style Truncate

Level limiters are also prefixed with $amavis_, but on the command line are specified with
the --limit option:

$amavis_SpamBlocked = 2

--limit SpamBlocked=2

The order of command line options and configuration file processing occurs as follows: 1)
The default configuration file is read if it exists and no --config_file was specified on
a command line. 2) Configuration files are read and processed in the order found on the
command line. 3) Command line options override any options already set either via command
line or from any configuration file.

Command line options are interpreted when they are seen on the command line, and later
options will override previously set options.

EXIT STATUS

The amavis-logwatch utility exits with a status code of 0, unless an error occurred, in
which case a non-zero exit status is returned.

EXAMPLES

Running Standalone
Note: amavis-logwatch reads its log data from one or more named Amavis log files, or from
STDIN. For brevity, where required, the examples below use the word file as the command
line argument meaning /path/to/amavis.log. Obviously you will need to substitute file
with the appropriate path.

To run amavis-logwatch in standalone mode, simply run:

amavis-logwatch file

A complete list of options and basic usage is available via:

amavis-logwatch --help

To print a summary only report of Amavis log data:

amavis-logwatch --detail 1 file

To produce a summary report and a one-level detail report for May 25th:

grep 'May 25' file | amavis-logwatch --detail 5

To produce only a top 10 list of Sent email domains, the summary report and detailed
reports are first disabled. Since commands line options are read and enabled left-to-
right, the Sent section is re-enabled to level 1 with a level 1 top 10 limiter:

amavis-logwatch --nosummary --nodetail \
--limit spamblocked '1 1:10:' file

The following command and its sample output shows a more complex level limiter example.
The command gives the top 4 spam blocked recipients (level 1), and under with each
recipient the top 2 sending IPs (level 2) and finally below that, only envelope from
addresses (level 3) with hit counts greater than 6. Ellipses indicate top N or threshold-
limited data:

amavis-logwatch --nosummary --nodetail \
--limit spamblocked '1:4: 2:2: 3::6' file

19346 Spam blocked -----------------------------------
756 [email protected]
12 10.0.0.1
12 <>
12 10.99.99.99
12 <>
...
640 [email protected]
8 10.0.0.1
8 <>
8 192.168.3.19
8 <>
...
595 [email protected]
8 10.0.0.1
8 <>
7 192.168.3.3
7 <>
...
547 [email protected]
8 192.168.3.19
8 <>
7 10.0.0.1
7 <>
...
...

Running within Logwatch
Note: Logwatch versions prior to 7.3.6, unless configured otherwise, required the --print
option to print to STDOUT instead of sending reports via email. Since version 7.3.6,
STDOUT is the default output destination, and the --print option has been replaced by
--output stdout. Check your configuration to determine where report output will be
directed, and add the appropriate option to the commands below.

To print a summary report for today's Amavis log data:

logwatch --service amavis --range today --detail 1

To print a report for today's Amavis log data, with one level
of detail in the Detailed section:

logwatch --service amavis --range today --detail 5

To print a report for yesterday, with two levels of detail in the Detailed section:

logwatch --service amavis --range yesterday --detail 6

To print a report from Dec 12th through Dec 14th, with four levels of detail in the
Detailed section:

logwatch --service amavis --range \
'between 12/12 and 12/14' --detail 8

To print a report for today, with all levels of detail:

logwatch --service amavis --range today --detail 10

Same as above, but leaves long lines uncropped:

logwatch --service amavis --range today --detail 11

Amavis Log Level
Amavis provides additional log information when the variable $log_level is increased above
the default 0 value. This information is used by the amavis-logwatch utility to provide
additional reports, not available with the default $log_level=0 value. A $log_level of 2
is suggested.

If you prefer not to increase the noise level in your main mail or Amavis logs, you can
configure syslog to log Amavis' output to multiple log files, where basic log entries are
routed to your main mail log(s) and more detailed entries routed to an Amavis-specific log
file used to feed the amavis-logwatch utility.

A convenient way to accomplish this is to change the Amavis configuration variables in
amavisd.conf as shown below:

amavisd.conf:
$log_level = 2;
$syslog_facility = 'local5';
$syslog_priority = 'debug';

This increases $log_level to 2, and sends Amavis' log entries to an alternate syslog
facility (eg. local5, user), which can then be routed to one or more log files, including
your main mail log file:

syslog.conf:
#mail.info -/var/log/maillog
mail.info;local5.notice -/var/log/maillog

local5.info -/var/log/amavisd-info.log

Amavis' typical $log_level 0 messages will be directed to both your maillog and to the
amavisd-info.log file, but higher $log_level messages will only be routed to the amavisd-
info.log file. For additional information on Amavis' logging, search the file
RELEASE_NOTES in the Amavis distribution for:

"syslog priorities are now dynamically derived"

ENVIRONMENT

The amavis-logwatch program uses the following (automatically set) environment variables
when running under Logwatch:

LOGWATCH_DETAIL_LEVEL
This is the detail level specified with the Logwatch command line argument --detail
or the Detail setting in the ...conf/services/amavis.conf configuration file.

LOGWATCH_DEBUG
This is the debug level specified with the Logwatch command line argument --debug.

amavis_xxx
The Logwatch program passes all settings amavis_xxx in the configuration file
...conf/services/amavis.conf to the amavis filter (which is actually named
.../scripts/services/amavis) via environment variable.

Use amavis-logwatch online using onworks.net services