duc-nox - Online in the Cloud

PROGRAM:

NAME

duc‐nox - index, query and graph disk usage

SYNOPSIS

duc‐nox <subcommand> [options]

DESCRIPTION

Duc is a collection of tools for inspecting and visualizing disk usage.

Duc maintains an indexed database of accumulated sizes of directories of your file system,
and allows you to query this database with some tools, or create graphs to show you where
your bytes are.

Duc scales quite well, it has been tested on systems with more then 500 million files and
several petabytes of storage.

USAGE

Duc comes with a command line tool called `duc‐nox`, which is used to create,

maintain and query the disk usage database.
run `duc‐nox help` to get a list of available commands. `duc‐nox help <subcommand>`
describes the usage of a specific subcommand. Run `duc‐nox help ‐‐all` for an
extensive list of all commands and their options.

Duc allows any option to be placed either on the command line or in a configuration file.
Options on the command line are preceded by a double-leading-dash (`‐‐option`), some
options have a corresponding short option which can be used as well with a single leading
dash. (`‐o`)

At startup duc tries to read its configuration from three locations in this particular
order: `/etc/ducrc`, `~/.ducrc` and `./.ducrc`.

A configuration file consists of sections and parameters. The section names correspond to
the duc subcommands for which the parameters in that section apply. A section begins with
the name of the section in square brackets and continues until the next section
begins.Sections contain parameters, one per line, which consist of a single option name
for boolean flags, or a option name and a value for options which take a value. See the
EXAMPLES section for an example of the configuration file format.

CREATING THE INDEX

Duc needs an index file of the file system before it is able to show any

information.
To create the index, run the `duc‐nox index` command. For example, to create an
index of your home directory run `duc‐nox index ~`

$ duc‐nox index /usr
Skipping lost+found: Permission denied
Indexed 333823 files and 48200 directories, (35.0GB total) in 1 seconds

The default location of the database is `$HOME/.duc.db`. To use a different database
location, use the DUC_DATABASE environment variable or specify the database location with
the ‐‐database argument.

You can run `duc‐ index` at any time later to rebuild the index.

QUERYING THE INDEX

Duc has various subcommands for querying or exploring the index:

· `duc‐nox info` shows a list of available directory trees in the database, and the time
and date of the last scan.

· `duc‐ ls` lists all files and directories under the given path on the console.

· `duc‐ ui` runs a ncurses based console user interface for exploring the file system
usage

OPTIONS

This section list all available subcommands and describes their usage and options. Global
options These options apply to all Duc subcommands:

· ‐‐debug: increase verbosity to debug level

· ‐h, ‐‐help: show help

· ‐q, ‐‐quiet: quiet mode, do not print any warning

· ‐v, ‐‐verbose: increase verbosity

· ‐‐version: output version information and exit

duc‐nox help Options for command `duc‐ help [options]`:

· ‐a, ‐‐all: show complete help for all commands

duc‐ index The 'index' subcommand performs a recursive scan of the given paths on the
filesystem and calculates the inclusive size of all directories. The results are written
to the index, and can later be queried by one of the other duc‐nox tools.

Options for command `duc‐nox index [options] PATH ...`:

· ‐b, ‐‐bytes: show file size in exact number of bytes

· ‐d, ‐‐database=VAL: use database file ARG

· ‐e, ‐‐exclude=VAL: exclude files matching ARG

· ‐H, ‐‐check‐hard‐links: count hard links only once. if two or more hard
links point to the same file, only one of the hard links is displayed and
counted

· ‐f, ‐‐force: force writing in case of corrupted db

· ‐‐hide‐file‐names: hide file names in index (privacy). the names of
directories will be preserved, but the names of the individual files will
be hidden

· ‐m, ‐‐max‐depth=VAL: limit directory names to given depth. when this
option is given duc will traverse the complete file system, but will only
the first VAL levels of directories in the database to reduce the size of
the index

· ‐x, ‐‐one‐file‐system: skip directories on different file systems

· ‐p, ‐‐progress: show progress during indexing

· ‐‐uncompressed: do not use compression for database. Duc enables
compression if the underlying database supports this. This reduces index
size at the cost of slightly longer indexing time

duc‐nox info Options for command `duc‐nox info [options]`:

· ‐a, ‐‐apparent: show apparent instead of actual file size

· ‐b, ‐‐bytes: show file size in exact number of bytes

· ‐d, ‐‐database=VAL: select database file to use [~/.duc.db]

duc‐nox ls The 'ls' subcommand queries the duc database and lists the inclusive size of
all files and directories on the given path. If no path is given the current working
directory is listed.

Options for command `duc‐nox ls [options] [PATH]`:

· ‐a, ‐‐apparent: show apparent instead of actual file size

· ‐‐ascii: use ASCII characters instead of UTF-8 to draw tree

· ‐b, ‐‐bytes: show file size in exact number of bytes

· ‐F, ‐‐classify: append file type indicator (one of */) to entries

· ‐c, ‐‐color: colorize output (only on ttys)

· ‐d, ‐‐database=VAL: select database file to use [~/.duc.db]

· ‐‐dirs‐only: list only directories, skip individual files

· `[hy]g`, `‐‐graph`: draw graph with relative size for each entry

· ‐l, ‐‐levels=VAL: traverse up to ARG levels deep [4]

· ‐R, ‐‐recursive: list subdirectories in a recursive tree view

duc‐nox xml Options for command `duc‐nox xml [options] [PATH]`:

· ‐d, ‐‐database=VAL: select database file to use [~/.duc.db]

· ‐x, ‐‐exclude‐files: exclude file from xml output, only include directories

· ‐s, ‐‐min_size=VAL: specify min size for files or directories

duc‐nox cgi Options for command `duc cgi [options] [PATH]`:

· ‐a, ‐‐apparent: Show apparent instead of actual file size

· ‐b, ‐‐bytes: show file size in exact number of bytes

· ‐‐css‐url=VAL: url of CSS style sheet to use instead of default CSS

· ‐d, ‐‐database=VAL: select database file to use [~/.duc.db]

· ‐l, ‐‐levels=VAL: draw up to ARG levels deep [4]

· ‐‐list: generate table with file list

· ‐‐tooltip: enable tooltip when hovering over the graph. enabling the tooltip
will cause an asynchronous HTTP request every time the mouse is moved and can
greatly increas the HTTP traffic to the web server

duc‐nox ui The 'ui' subcommand queries the duc database and runs an interactive ncurses
utility for exploring the disk usage of the given path. If no path is given the current
working directory is explored.

The following keys can be used to navigate and alter the file system:

up, pgup, j: move cursor up
down, pgdn, k: move cursor down
left, backspace: go up to parent directory (..)
right, enter: descent into selected directory
a: toggle between actual and apparent disk usage
b: toggle between exact and abbreviated sizes
c: toggle between color and monochrome display
g: toggle graph
h: show help. press 'q' to return to the main screen
q, escape: quit

Options for command `duc‐nox ui [options] [PATH]`:

· ‐a, ‐‐apparent: show apparent instead of actual file size

· ‐b, ‐‐bytes: show file size in exact number of bytes

· ‐c, ‐‐color: colorize output

· ‐d, ‐‐database=VAL: select database file to use [~/.duc.db]

CGI INTERFACING

The `duc‐nox` binary has support for a rudimentary CGI interface,

currently only tested with apache.
The CGI interface generates a simple HTML page with a list of indexed directories,
and shows a clickable graph for navigating the file system. If the option `‐‐list`
is given, a list of top sized files/dirs is also written.

Configuration is done by creating a simple shell script as .cgi in a directory which is
configured for CGI execution by your web server (usually `/usr/lib/cgi-bin`). The shell
script should simply start duc‐nox, and pass the location of the database to navigate.

An example duc.cgi script would be

#!/bin/sh
/usr/bin/duc‐nox cgi -d /home/jenny/.duc.db

* Make sure the database file is readable by the user (usually www‐data)
* Debugging is best done by inspecting the web server's error log
* Make sure the .cgi script has execute permissions (`chmod +x duc.cgi`)

Some notes:

· The HTML page is generated with a simple embedded CSS style sheet. If the style
is not to your liking you can provide an external CSS url with the ‐‐css‐url
option which will then be used instead of the embedded style definition.

· Add the option ‐‐list to generate a table of top sized files and directories in
the HTML page.

The current CGI configuration is not very flexible, nor secure. It is not advised to run
the CGI from public reachable web servers, use at your own risk.

A NOTE ON FILE SIZE AND DISK USAGE

The concepts of 'file size' and 'disk usage' can be a bit confusing. Files on disk have an
apparent size, which indicates how much bytes are in the file from the users point of
view; this is the size reported by tools like `ls ‐l`.

The apparent size can be any number, from 0 bytes up to several TB.
The actual number of bytes which are used on the filesystem to store the file can
differ from this apparent size for a number of reasons: disks store data in blocks,
which cause files to always take up a multiple of the block size, files can have
holes ('sparse' files), and other technical reasons. This number is always a
multiple of 512, which means that the actual size used for a file is almost always
a bit more then its apparent size.

Duc has two modes for counting file sizes:

‐ `apparent size`: this is the size as reported by `ls`. This number indicates the file
length, which is usually smaller then the actual disk usage.

‐ `actual size`: this is the size as reported by `du` and `df`. The actual file size tells
you how much disk is actually used by a file, and is always a multiple of 512 bytes.

The default mode used by duc‐nox is to use the 'actual size'. Most duc‐nox commands to
report disk usage (`duc‐nox ls`, etc) have an option to change between these two modes
(usually the `‐a`).

EXAMPLES

Index the /usr directory, writing to the default database location ~/.duc.db:

$ duc‐nox index /usr

List all files and directories under /usr/local, showing relative file sizes in a graph:

$ duc‐nox ls ‐Fg /usr/local
4.7G lib/ [+++++++++++++++++++++++++++++++++++++++++++]
3.1G share/ [++++++++++++++++++++++++++++ ]
2.7G src/ [++++++++++++++++++++++++ ]
814.9M bin/ [+++++++ ]
196.6M include/ [+ ]
66.6M x86_64-w64-mingw32/ [ ]
59.9M local/ [ ]
38.8M i686-w64-mingw32/ [ ]
20.3M sbin/ [ ]
13.6M lib32/ [ ]
13.3M libx32/ [ ]

or use the ‐R
options for the tree view:

$ duc‐nox ls ‐RF /etc/logcheck
24.0K `+‐ ignore.d.server/
4.0K | `+‐ hddtemp
4.0K | |‐ ntpdate
4.0K | |‐ lirc
4.0K | |‐ rsyslog
4.0K | `‐ libsasl2‐modules
8.0K |‐ ignore.d.workstation/
4.0K | `‐ lirc
8.0K `‐ ignore.d.paranoid/
4.0K `‐ lirc

The following sample configuration file defines default parameters for the `duc‐nox ls`
command and defines a global option to configure the database path which is used by all
subcommands

[global]
database /var/cache/duc.db

[ls]
recursive
classify
color

[ui]
color

Use duc-nox online using onworks.net services