agedu - Online in the Cloud

PROGRAM:

NAME

agedu - correlate disk usage with last-access times to identify large and disused data

SYNOPSIS

agedu [ options ] action [action...]

DESCRIPTION

agedu scans a directory tree and produces reports about how much disk space is used in
each directory and subdirectory, and also how that usage of disk space corresponds to
files with last-access times a long time ago.

In other words, agedu is a tool you might use to help you free up disk space. It lets you
see which directories are taking up the most space, as du does; but unlike du, it also
distinguishes between large collections of data which are still in use and ones which have
not been accessed in months or years - for instance, large archives downloaded, unpacked,
used once, and never cleaned up. Where du helps you find what's using your disk space,
agedu helps you find what's wasting your disk space.

agedu has several operating modes. In one mode, it scans your disk and builds an index
file containing a data structure which allows it to efficiently retrieve any information
it might need. Typically, you would use it in this mode first, and then run it in one of a
number of `query' modes to display a report of the disk space usage of a particular
directory and its subdirectories. Those reports can be produced as plain text (much like
du) or as HTML. agedu can even run as a miniature web server, presenting each directory's
HTML report with hyperlinks to let you navigate around the file system to similar reports
for other directories.

So you would typically start using agedu by telling it to do a scan of a directory tree
and build an index. This is done with a command such as

$ agedu -s /home/fred

which will build a large data file called agedu.dat in your current directory. (If that
current directory is inside /home/fred, don't worry - agedu is smart enough to discount
its own index file.)

Having built the index, you would now query it for reports of disk space usage. If you
have a graphical web browser, the simplest and nicest way to query the index is by running
agedu in web server mode:

$ agedu -w

which will print (among other messages) a URL on its standard output along the lines of

URL: http://127.0.0.1:48638/

(That URL will always begin with `127.', meaning that it's in the localhost address space.
So only processes running on the same computer can even try to connect to that web server,
and also there is access control to prevent other users from seeing it - see below for
more detail.)

Now paste that URL into your web browser, and you will be shown a graphical representation
of the disk usage in /home/fred and its immediate subdirectories, with varying colours
used to show the difference between disused and recently-accessed data. Click on any
subdirectory to descend into it and see a report for its subdirectories in turn; click on
parts of the pathname at the top of any page to return to higher-level directories. When
you've finished browsing, you can just press Ctrl-D to send an end-of-file indication to
agedu, and it will shut down.

After that, you probably want to delete the data file agedu.dat, since it's pretty large.
In fact, the command agedu -R will do this for you; and you can chain agedu commands on
the same command line, so that instead of the above you could have done

$ agedu -s /home/fred -w -R

for a single self-contained run of agedu which builds its index, serves web pages from it,
and cleans it up when finished.

If you don't have a graphical web browser, you can do text-based queries as well. Having
scanned /home/fred as above, you might run

$ agedu -t /home/fred

which again gives a summary of the disk usage in /home/fred and its immediate
subdirectories; but this time agedu will print it on standard output, in much the same
format as du. If you then want to find out how much old data is there, you can add the -a
option to show only files last accessed a certain length of time ago. For example, to show
only files which haven't been looked at in six months or more:

$ agedu -t /home/fred -a 6m

That's the essence of what agedu does. It has other modes of operation for more complex
situations, and the usual array of configurable options. The following sections contain a
complete reference for all its functionality.

OPERATING MODES

This section describes the operating modes supported by agedu. Each of these is in the
form of a command-line option, sometimes with an argument. Multiple operating-mode options
may appear on the command line, in which case agedu will perform the specified actions one
after another. For instance, as shown in the previous section, you might want to perform a
disk scan and immediately launch a web server giving reports from that scan.

-s directory or --scan directory
In this mode, agedu scans the file system starting at the specified directory, and
indexes the results of the scan into a large data file which other operating modes
can query.

By default, the scan is restricted to a single file system (since the expected use
of agedu is that you would probably use it because a particular disk partition was
running low on space). You can remove that restriction using the --cross-fs option;
other configuration options allow you to include or exclude files or entire
subdirectories from the scan. See the next section for full details of the
configurable options.

The index file is created with restrictive permissions, in case the file system you
are scanning contains confidential information in its structure.

Index files are dependent on the characteristics of the CPU architecture you
created them on. You should not expect to be able to move an index file between
different types of computer and have it continue to work. If you need to transfer
the results of a disk scan to a different kind of computer, see the -D and -L
options below.

-w or --web
In this mode, agedu expects to find an index file already written. It allocates a
network port, and starts up a web server on that port which serves reports
generated from the index file. By default it invents its own URL and prints it out.

The web server runs until agedu receives an end-of-file event on its standard
input. (The expected usage is that you run it from the command line, immediately
browse web pages until you're satisfied, and then press Ctrl-D.) To disable the EOF
behaviour, use the --no-eof option.

In case the index file contains any confidential information about your file
system, the web server protects the pages it serves from access by other people. On
Linux, this is done transparently by means of using /proc/net/tcp to check the
owner of each incoming connection; failing that, the web server will require a
password to view the reports, and agedu will print the password it invented on
standard output along with the URL.

Configurable options for this mode let you specify your own address and port number
to listen on, and also specify your own choice of authentication method (including
turning authentication off completely) and a username and password of your choice.

-t directory or --text directory
In this mode, agedu generates a textual report on standard output, listing the disk
usage in the specified directory and all its subdirectories down to a given depth.
By default that depth is 1, so that you see a report for directory itself and all
of its immediate subdirectories. You can configure a different depth (or no depth
limit) using -d, described in the next section.

Used on its own, -t merely lists the total disk usage in each subdirectory; agedu's
additional ability to distinguish unused from recently-used data is not activated.
To activate it, use the -a option to specify a minimum age.

The directory structure stored in agedu's index file is treated as a set of literal
strings. This means that you cannot refer to directories by synonyms. So if you ran
agedu -s ., then all the path names you later pass to the -t option must be either
`.' or begin with `./'. Similarly, symbolic links within the directory you scanned
will not be followed; you must refer to each directory by its canonical, symlink-
free pathname.

-R or --remove
In this mode, agedu deletes its index file. Running just agedu -R on its own is
therefore equivalent to typing rm agedu.dat. However, you can also put -R on the
end of a command line to indicate that agedu should delete its index file after it
finishes performing other operations.

-D or --dump
In this mode, agedu reads an existing index file and produces a dump of its
contents on standard output. This dump can later be loaded into a new index file,
perhaps on another computer.

-L or --load
In this mode, agedu expects to read a dump produced by the -D option from its
standard input. It constructs an index file from that dump, exactly as it would
have if it had read the same data from a disk scan in -s mode.

-S directory or --scan-dump directory
In this mode, agedu will scan a directory tree and convert the results straight
into a dump on standard output, without generating an index file at all. So running
agedu -S /path should produce equivalent output to that of agedu -s /path -D,
except that the latter will produce an index file as a side effect whereas -S will
not.

(The output will not be exactly identical, due to a difference in treatment of
last-access times on directories. However, it should be effectively equivalent for
most purposes. See the documentation of the --dir-atime option in the next section
for further detail.)

-H directory or --html directory
In this mode, agedu will generate an HTML report of the disk usage in the specified
directory and its immediate subdirectories, in the same form that it serves from
its web server in -w mode.

By default, a single HTML report will be generated and simply written to standard
output, with no hyperlinks pointing to other similar pages. If you also specify the
-d option (see below), agedu will instead write out a collection of HTML files with
hyperlinks between them, and call the top-level file index.html.

--cgi In this mode, agedu will run as the bulk of a CGI script which provides the same
set of web pages as the built-in web server would. It will read the usual CGI
environment variables, and write CGI-style data to its standard output.

The actual CGI program itself should be a tiny wrapper around agedu which passes it
the --cgi option, and also (probably) -f to locate the index file. agedu will do
everything else.

No access control is performed in this mode: restricting access to CGI scripts is
assumed to be the job of the web server.

-h or --help
Causes agedu to print some help text and terminate immediately.

-V or --version
Causes agedu to print its version number and terminate immediately.

OPTIONS

This section describes the various configuration options that affect agedu's operation in
one mode or another.

The following option affects nearly all modes (except -S):

-f filename or --file filename
Specifies the location of the index file which agedu creates, reads or removes
depending on its operating mode. By default, this is simply `agedu.dat', in
whatever is the current working directory when you run agedu.

The following options affect the disk-scanning modes, -s and -S:

--cross-fs and --no-cross-fs
These configure whether or not the disk scan is permitted to cross between
different file systems. The default is not to: agedu will normally skip over
subdirectories on which a different file system is mounted. This makes it
convenient when you want to free up space on a particular file system which is
running low. However, in other circumstances you might wish to see general
information about the use of space no matter which file system it's on (for
instance, if your real concern is your backup media running out of space, and if
your backups do not treat different file systems specially); in that situation, use
--cross-fs.

(Note that this default is the opposite way round from the corresponding option in
du.)

--prune wildcard and --prune-path wildcard
These cause particular files or directories to be omitted entirely from the scan.
If agedu's scan encounters a file or directory whose name matches the wildcard
provided to the --prune option, it will not include that file in its index, and
also if it's a directory it will skip over it and not scan its contents.

Note that in most Unix shells, wildcards will probably need to be escaped on the
command line, to prevent the shell from expanding the wildcard before agedu sees
it.

--prune-path is similar to --prune, except that the wildcard is matched against the
entire pathname instead of just the filename at the end of it. So whereas --prune
*a*b* will match any file whose actual name contains an a somewhere before a b,
--prune-path *a*b* will also match a file whose name contains b and which is inside
a directory containing an a, or any file inside a directory of that form, and so
on.

--exclude wildcard and --exclude-path wildcard
These cause particular files or directories to be omitted from the index, but not
from the scan. If agedu's scan encounters a file or directory whose name matches
the wildcard provided to the --exclude option, it will not include that file in its
index - but unlike --prune, if the file in question is a directory it will still
scan its contents and index them if they are not ruled out themselves by --exclude
options.

As above, --exclude-path is similar to --exclude, except that the wildcard is
matched against the entire pathname.

--include wildcard and --include-path wildcard
These cause particular files or directories to be re-included in the index and the
scan, if they had previously been ruled out by one of the above exclude or prune
options. You can interleave include, exclude and prune options as you wish on the
command line, and if more than one of them applies to a file then the last one
takes priority.

For example, if you wanted to see only the disk space taken up by MP3 files, you
might run

$ agedu -s . --exclude '*' --include '*.mp3'

which will cause everything to be omitted from the scan, but then the MP3 files to
be put back in. If you then wanted only a subset of those MP3s, you could then
exclude some of them again by adding, say, `--exclude-path './queen/*'' (or, more
efficiently, `--prune ./queen') on the end of that command.

As with the previous two options, --include-path is similar to --include except
that the wildcard is matched against the entire pathname.

--progress, --no-progress and --tty-progress
When agedu is scanning a directory tree, it will typically print a one-line
progress report every second showing where it has reached in the scan, so you can
have some idea of how much longer it will take. (Of course, it can't predict
exactly how long it will take, since it doesn't know which of the directories it
hasn't scanned yet will turn out to be huge.)

By default, those progress reports are displayed on agedu's standard error channel,
if that channel points to a terminal device. If you need to manually enable or
disable them, you can use the above three options to do so: --progress
unconditionally enables the progress reports, --no-progress unconditionally
disables them, and --tty-progress reverts to the default behaviour which is
conditional on standard error being a terminal.

--dir-atime and --no-dir-atime
In normal operation, agedu ignores the atimes (last access times) on the
directories it scans: it only pays attention to the atimes of the files inside
those directories. This is because directory atimes tend to be reset by a lot of
system administrative tasks, such as cron jobs which scan the file system for one
reason or another - or even other invocations of agedu itself, though it tries to
avoid modifying any atimes if possible. So the literal atimes on directories are
typically not representative of how long ago the data in question was last accessed
with real intent to use that data in particular.

Instead, agedu makes up a fake atime for every directory it scans, which is equal
to the newest atime of any file in or below that directory (or the directory's last
modification time, whichever is newest). This is based on the assumption that all
important accesses to directories are actually accesses to the files inside those
directories, so that when any file is accessed all the directories on the path
leading to it should be considered to have been accessed as well.

In unusual cases it is possible that a directory itself might embody important data
which is accessed by reading the directory. In that situation, agedu's atime-faking
policy will misreport the directory as disused. In the unlikely event that such
directories form a significant part of your disk space usage, you might want to
turn off the faking. The --dir-atime option does this: it causes the disk scan to
read the original atimes of the directories it scans.

The faking of atimes on directories also requires a processing pass over the index
file after the main disk scan is complete. --dir-atime also turns this pass off.
Hence, this option affects the -L option as well as -s and -S.

(The previous section mentioned that there might be subtle differences between the
output of agedu -s /path -D and agedu -S /path. This is why. Doing a scan with -s
and then dumping it with -D will dump the fully faked atimes on the directories,
whereas doing a scan-to-dump with -S will dump only partially faked atimes -
specifically, each directory's last modification time - since the subsequent
processing pass will not have had a chance to take place. However, loading either
of the resulting dump files with -L will perform the atime-faking processing pass,
leading to the same data in the index file in each case. In normal usage it should
be safe to ignore all of this complexity.)

--mtime
This option causes agedu to index files by their last modification time instead of
their last access time. You might want to use this if your last access times were
completely useless for some reason: for example, if you had recently searched every
file on your system, the system would have lost all the information about what
files you hadn't recently accessed before then. Using this option is liable to be
less effective at finding genuinely wasted space than the normal mode (that is, it
will be more likely to flag things as disused when they're not, so you will have
more candidates to go through by hand looking for data you don't need), but may be
better than nothing if your last-access times are unhelpful.

Another use for this mode might be to find recently created large data. If your
disk has been gradually filling up for years, the default mode of agedu will let
you find unused data to delete; but if you know your disk had plenty of space
recently and now it's suddenly full, and you suspect that some rogue program has
left a large core dump or output file, then agedu --mtime might be a convenient way
to locate the culprit.

The following option affects all the modes that generate reports: the web server mode -w,
the stand-alone HTML generation mode -H and the text report mode -t.

--files
This option causes agedu's reports to list the individual files in each directory,
instead of just giving a combined report for everything that's not in a
subdirectory.

The following option affects the text report mode -t.

-a age or --age age
This option tells agedu to report only files of at least the specified age. An age
is specified as a number, followed by one of `y' (years), `m' (months), `w' (weeks)
or `d' (days). (This syntax is also used by the -r option.) For example, -a 6m will
produce a text report which includes only files at least six months old.

The following options affect the stand-alone HTML generation mode -H and the text report
mode -t.

-d depth or --depth depth
This option controls the maximum depth to which agedu recurses when generating a
text or HTML report.

In text mode, the default is 1, meaning that the report will include the directory
given on the command line and all of its immediate subdirectories. A depth of two
includes another level below that, and so on; a depth of zero means only the
directory on the command line.

In HTML mode, specifying this option switches agedu from writing out a single HTML
file to writing out multiple files which link to each other. A depth of 1 means
agedu will write out an HTML file for the given directory and also one for each of
its immediate subdirectories.

If you want agedu to recurse as deeply as possible, give the special word `max' as
an argument to -d.

-o filename or --output filename
This option is used to specify an output file for agedu to write its report to. In
text mode or single-file HTML mode, the argument is treated as the name of a file.
In multiple-file HTML mode, the argument is treated as the name of a directory: the
directory will be created if it does not already exist, and the output HTML files
will be created inside it.

The following options affect the web server mode -w, and in some cases also the stand-
alone HTML generation mode -H:

-r age range or --age-range age range
The HTML reports produced by agedu use a range of colours to indicate how long ago
data was last accessed, running from red (representing the most disused data) to
green (representing the newest). By default, the lengths of time represented by the
two ends of that spectrum are chosen by examining the data file to see what range
of ages appears in it. However, you might want to set your own limits, and you can
do this using -r.

The argument to -r consists of a single age, or two ages separated by a minus sign.
An age is a number, followed by one of `y' (years), `m' (months), `w' (weeks) or
`d' (days). (This syntax is also used by the -a option.) The first age in the range
represents the oldest data, and will be coloured red in the HTML; the second age
represents the newest, coloured green. If the second age is not specified, it will
default to zero (so that green means data which has been accessed just now).

For example, -r 2y will mark data in red if it has been unused for two years or
more, and green if it has been accessed just now. -r 2y-3m will similarly mark data
red if it has been unused for two years or more, but will mark it green if it has
been accessed three months ago or later.

--address addr[:port]
Specifies the network address and port number on which agedu should listen when
running its web server. If you want agedu to listen for connections coming in from
any source, specify the address as the special value ANY. If the port number is
omitted, an arbitrary unused port will be chosen for you and displayed.

If you specify this option, agedu will not print its URL on standard output (since
you are expected to know what address you told it to listen to).

--auth auth-type
Specifies how agedu should control access to the web pages it serves. The options
are as follows:

magic This option only works on Linux, and only when the incoming connection is
from the same machine that agedu is running on. On Linux, the special file
/proc/net/tcp contains a list of network connections currently known to the
operating system kernel, including which user id created them. So agedu will
look up each incoming connection in that file, and allow access if it comes
from the same user id under which agedu itself is running. Therefore, in
agedu's normal web server mode, you can safely run it on a multi-user
machine and no other user will be able to read data out of your index file.

basic In this mode, agedu will use HTTP Basic authentication: the user will have
to provide a username and password via their browser. agedu will normally
make up a username and password for the purpose, but you can specify your
own; see below.

none In this mode, the web server is unauthenticated: anyone connecting to it has
full access to the reports generated by agedu. Do not do this unless there
is nothing confidential at all in your index file, or unless you are certain
that nobody but you can run processes on your computer.

default
This is the default mode if you do not specify one of the above. In this
mode, agedu will attempt to use Linux magic authentication, but if it
detects at startup time that /proc/net/tcp is absent or non-functional then
it will fall back to using HTTP Basic authentication and invent a user name
and password.

--auth-file filename or --auth-fd fd
When agedu is using HTTP Basic authentication, these options allow you to specify
your own user name and password. If you specify --auth-file, these will be read
from the specified file; if you specify --auth-fd they will instead be read from a
given file descriptor which you should have arranged to pass to agedu. In either
case, the authentication details should consist of the username, followed by a
colon, followed by the password, followed immediately by end of file (no trailing
newline, or else it will be considered part of the password).

--title title
Specify the string that appears at the start of the <title> section of the output
HTML pages. The default is `agedu'. This title is followed by a colon and then the
path you're viewing within the index file. You might use this option if you were
serving agedu reports for several different servers and wanted to make it clearer
which one a user was looking at.

--no-eof
Stop agedu in web server mode from looking for end-of-file on standard input and
treating it as a signal to terminate.

LIMITATIONS

The data file is pretty large. The core of agedu is the tree-based data structure it uses
in its index in order to efficiently perform the queries it needs; this data structure
requires O(N log N) storage. This is larger than you might expect; a scan of my own home
directory, containing half a million files and directories and about 20Gb of data,
produced an index file over 60Mb in size. Furthermore, since the data file must be memory-
mapped during most processing, it can never grow larger than available address space, so a
really big filesystem may need to be indexed on a 64-bit computer. (This is one reason for
the existence of the -D and -L options: you can do the scanning on the machine with access
to the filesystem, and the indexing on a machine big enough to handle it.)

The data structure also does not usefully permit access control within the data file, so
it would be difficult - even given the willingness to do additional coding - to run a
system-wide agedu scan on a cron job and serve the right subset of reports to each user.

In certain circumstances, agedu can report false positives (reporting files as disused
which are in fact in use) as well as the more benign false negatives (reporting files as
in use which are not). This arises when a file is, semantically speaking, `read' without
actually being physically read. Typically this occurs when a program checks whether the
file's mtime has changed and only bothers re-reading it if it has; programs which do this
include rsync(1) and make(1). Such programs will fail to update the atime of unmodified
files despite depending on their continued existence; a directory full of such files will
be reported as disused by agedu even in situations where deleting them will cause trouble.

Finally, of course, agedu's normal usage mode depends critically on the OS providing last-
access times which are at least approximately right. So a file system mounted with Linux's
`noatime' option, or the equivalent on any other OS, will not give useful results!
(However, the Linux mount option `relatime', which distributions now tend to use by
default, should be fine for all but specialist purposes: it reduces the accuracy of last-
access times so that they might be wrong by up to 24 hours, but if you're looking for
files that have been unused for months or years, that's not a problem.)

LICENCE

agedu is free software, distributed under the MIT licence. Type agedu --licence to see the
full licence text.

Use agedu online using onworks.net services