rsbackup - Online in the Cloud

PROGRAM:

NAME

rsbackup - rsync-based backup utility

SYNOPSIS

rsbackup [OPTIONS] [--] [SELECTOR...]
rsbackup --retire [OPTIONS] [--] [SELECTOR...]
rsbackup --retire-device [OPTIONS] [--] DEVICE...

DESCRIPTION

Backs up files from one or more (remote) destinations to a single backup storage
directory, preserving their contents, layout, ownership, permissions, timestamps and
hardlink structure.

Incremental backups are achieved by hard-linking identical files within successive backups
of the same files.

OPTIONS

Action Options
At least one of these options must be specified. When multiple actions are specified,
they are executed in the order shown below.

--backup, -b
Make a backup of the selected volumes. At most one backup of a given volume will
be made per day.

--retire-device
Retire the named devices. Retiring a device only means deleting the records of it.
Files on the device itself are not touched.

If the device is still listed in the configuration file then you will be asked
whether you really want to retire it; you can suppress this check with the --force
option.

--retire
Retire the named hosts and volumes. Retiring a volume means deleting any available
backups for the volume and the records of them. Records corresponding to backups
on unavailable devices are not removed.

If you just want to remove backup records for retired volumes but want to keep the
backups, you should either manually remove the records (see SCHEMA below), or
rename it within the volume.

If the volume is still listed in the configuration file then you will be asked
whether you really want to retire it; you can suppress this check with the --force
option.

--prune, -p
Prune old backups of selected volumes. See PRUNING below.

--prune-incomplete,-P
Prune incomplete backups of selected volumes. Any backups that failed before
completion will be removed.

--html PATH, -H PATH
Write an HTML report to PATH. The report covers all volumes, not just selected
ones. PATH can be - to write to standard output.

--text PATH, -T PATH
Write a plain text report to PATH. The report covers all volumes, not just
selected ones. PATH can be - to write to standard output.

--email ADDRESS, -e ADDRESS
Email a report to ADDRESS. The contents is equivalent to the output of --text and
--html.

--dump-config
Writes the parsed configuration file to standard output. Must not be combined with
any other action option.

With --verbose, the configuration file is annotated with descriptive comments.

General Options
--config PATH, -c PATH
The path to the configuration file. The default is /etc/rsbackup/config.

--store PATH, -s PATH
Specify the destination directory to back up to. Using this option (possibly more
than once) is equivalent to removing the store directives from the configuration
file and replacing them with the paths give in --store options.

This option implicitly enables the --warn-store option.

--verbose, -v
Enable verbose mode. Various messages will be displayed to report progress and the
rsync --quiet option is suppressed.

--dry-run, -n
Enable dry-run mode. Commands will be displayed but nothing will actually be done.

--force, -f
Suppress checks made when retiring devices and volumes.

--wait, -w
Waits rather than giving up if another copy of rsbackup is running.

--database, -D PATH
Override the path to the backup database.

--help, -h
Display a usage message.

--version, -V
Display the version number.

Report Verbosity
--logs VERBOSITY
Controls which logfiles for a given volume/device pair to include in the report.
The possible values of VERBOSITY are:

all Includes all nonempty logfiles, even if the backup succeeded.

errors Includes all error logfiles.

recent Includes only the most recent error logfile.

latest Includes only the latest logfile, even if the backup succeeded.

failed Includes only the most recent logfile but only if that attempt failed. This
is the default.

Warning Options
--warn-unknown
Display warnings for unknown devices, hosts and volumes. (Warnings will always be
included in the report, this refers to runtime error output.)

--warn-store
Display warnings for unsuitable store directories and unavailable devices.

--warn-unreachable
Display warnings for unreachable hosts.

--no-warn-partial
Suppress warnings for rsync "partial transfer" diagnostics (which are on by
default).

--warn-all, -W
Enable all --warn- options.

--no-errors
Suppress display of errors from rsync.

Volume Selection
The list of selectors on the command line determines what subset of the known volumes are
backed up, pruned or retired. The following selectors are possible:

HOST Select all volumes for the host.

HOST:VOLUME Select the volume.

-HOST Deselect all volumes for the host.

-HOST:VOLUME Deselect the volume.

* Select all volumes.

If no hosts or volumes are specified on the command line then all volumes are selected for
backing up or pruning. For retiring, you must explicitly select hosts or volumes to
retire and only positive selections are possible.

CONFIGURATION FILE

The config file contains global directives and a series of host stanzas. Each host stanze
in turn contains host directives and volume stanzas. Although it is not enforced it is
suggested that host and volume stanzas are indented.

Comments are introduced by an initial "#".

Command arguments may be quoted, using "double quotes". Quotes and backslashes within
quoted strings are escaped with backslashes.

Global Directives
Global directives control some general aspect of the program.

colors GOOD BAD
The colors used to represent good states (a recent backup) and bad states (no
sufficiently recent backup).

GOOD and BAD are integer values representing RGB triples. It is most convenient to
write them in hex, e.g. as 0xRRGGBB. For example, black is 0x000000, red is
0xFF0000, and so on.

device DEVICE
Names a device. This can be used multiple times. The store must have a file
called STORE/device-id which contains a known device name. Backups will only be
made to known devices.

When a device is lost or destroyed, remove its device entry and use the
--prune-unknown option to delete records of backups on it.

Device names may contain letters, digits, dots and underscores.

include PATH
Include another file as part of the configuration. If PATH is a directory then the
files within it are included (excluding dotfiles, backup and recovery files).

keep-prune-logs DAYS
The number of days to keep records of pruned backups for. The default is 31.

lock PATH
Enable locking. If this directive is present then PATH will be used as a lockfile
for operations that change anything (--backup, --prune, etc).

The lock is made by opening PATH and calling flock(2) on it with LOCK_EX.

logs PATH
The directory to store logfiles and backup records. The default is
/var/log/backup.

post-access-hook COMMAND...
A command to execute after all backup and prune operations. This is executed only
once per invocation of rsbackup. A backup is still considered to have succeeded
even if the post-access hook fails (i.e. exits nonzero). See HOOKS below.

pre-access-hook COMMAND...
A command to execute before anything that accesses any backup devices (i.e. backup
and prune operations). This is executed only once per invocation of rsbackup and
if it fails (i.e. exits nonzero) then rsbackup terminates immediately. See HOOKS
below.

public true|false
If true, backups are public. Normally backups must only be accessible by the
calling user. This option suppresses the check.

report-prune-logs DAYS
The number of days of pruning logs to put in hte report. The default is 3.

sendmail PATH
The path to the executable to use for sending email. The default is platform-
dependent but typically /usr/sbin/sendmail. The executable should support the -t,
-oee, -oi and -odb options.

store PATH
A path at which a backup device may be mounted. This can be used multiple times.

store-pattern PATTERN
A glob(7) pattern matching paths at which a backup device may be mounted. This can
be used multiple times.

stylesheet PATH
The path to the stylesheet to use in the HTML report. If this is absent then a
built-in default stylesheet is used.

Inheritable Directives
Inheritable directives control an aspect of one or more backups. They can be specified at
the global level or in a host or volume stanza (see below). If one appears in multiple
places then volume settings override host settings and host settings override global
settings.

hook-timeout SECONDS
How long to wait before concluding a hook has hung, in seconds. The default is 0,
which means to wait indefinitely.

max-age DAYS
The maximum age of the most recent backup before you feel uncomfortable. The
default is 3, meaning that if a volume hasn't been backed up in the last 3 days it
will have red ink in the HTML report.

min-backups COUNT
The minimum number of backups for each volume to keep on each store, when pruning.
The default is 1.

(This is an alias for prune-parameter min-backups and will be removed in a future
version.)

post-backup-hook COMMAND...
A command to execute after finishing a backup, or after it failed. A backup is
still considered to have succeeded even if the post-backup hook fails (exits
nonzero). See HOOKS below.

pre-backup-hook COMMAND...
A command to execute before starting a backup. If this hook fails (i.e. exits
nonzero) then the backup is not made and the post-backup hook will not be run. See
HOOKS below.

This hook can override the source path for the backup by writing a new source path
to standard output.

prune-age DAYS
The age at which a backup may be pruned. The default is 366, meaning a backup will
never be pruned until it is at least a whole year old.

(This is an alias for prune-parameter prune-age and will be removed in a future
version.)

prune-parameter NAME VALUE
Set a parameter for the pruning policy. See PRUNING below.

prune-parameter --remove NAME
Remove a parameter for pruning policy.

prune-policy NAME
The pruning policy to use. See PRUNING below.

rsync-timeout SECONDS
How long to wait before concluding rsync has hung, in seconds. The default is 0,
which means to wait indefinitely.

ssh-timeout SECONDS
How long to wait before concluding a host is down, in seconds. The default is 60.

Host Directives
A host stanza is started by a host directive.

host HOST
Introduce a host stanza. The name is used for the backup directory for this host.

The following directives, and volume stanzas (see below), can appear in a host stanza:

always-up true|false
If true, the host is expected to always be available. If it is not then a warning
will be issued when making a backup if it is not. Failed attempts to make a backup
will also be recorded as failures for always-up hosts (normally hosts that cannot
be reached are silently skipped).

devices PATTERN
A glob(3) pattern restricting the devices that this host will be backed up to.

Note that only backup creation honors this restriction. Pruning and retiring do
not.

hostname HOSTNAME
The SSH hostname for this host. The default is the name from the host stanza.

The hostname localhost is treated specially: it is assumed to always be identical
to the local system, so files will be read from the local filesystem.

priority INTEGER
The priority of this host. Hosts are backed up in descending priority order. The
default priority is 0.

user USERNAME
The SSH username for this host. The default is not to supply a username.

In addition, inheritable directives can appear in a host stanza, and override any
appearance of them at the global level.

Conventionally the contents of a host stanza are indented.

Remote hosts are accessed by SSH. The user rsbackup runs as must be able to connect to
the remote host (and without a password being entered if it is to be run from a cron job
or similar).

Volume Directives
A volume stanza is started by a volume directive.

volume VOLUME PATH
Introduce a volume stanza. The name is used for the backup directory for this
volume. The path is the absolute path on the host.

The following directives can appear in a volume stanza:

check-file PATH
Checks that PATH exists before backing up the volume. PATH may be either an
absolute path or a relative path (to the root of the volume). It need not be
inside the volume though the usual use would be to check for a file which is always
present there.

This check is done before executing the pre-backup-hook, so it applies to the real
path to the volume, not the rewritten path.

check-mounted true|false
If true, checks that the volume's path is a mount point before backing up the
volume.

This check is done before executing the pre-backup-hook, so it applies to the real
path to the volume, not the rewritten path.

Note that if multiple check- options are used, all checks must pass for the volume
to be backed up.

exclude PATTERN
An exclusion for this volume. The pattern is passed to the rsync --exclude option.
This directive may appear multiple times per volume.

See the rsync man page for full details.

traverse true|false
If true, traverse mount points. This suppresses the rsync --one-file-system
option.

In addition, inheritable directives can appear in a volume stanza, and override any
appearance of them at the host or global level.

Conventionally the contents of a volume stanza are indented.

PRUNING

This is process of removing old backups (using the --prune option). The pruning policy
used to determine which backups to remove is set with the inheritable prune-policy
directive, and parameters to the policy set via the prune-parameter directive.

The available policies are listed below. The default policy is age.

age
This policy deletes backups older than a minimum age, provided a minimum number of backups
on a device remain available. The following pruning parameters are supported:

min-backups
The minimum number of backups of the volume to maintain on the device. Pruning
will never cause the number of backups to fall below this value. The default (and
minimum) is 1.

prune-age
The age after backups become eligible for pruning, in days. Only backups more than
this many days old will be pruned. The default is 366 and the minimum is 1.

For backwards compatibility, these values can also be set using the directives of the same
name. This will be disabled in a future version.

decay
This policy thins out backups older than a minimum age, using a configurable decay pattern
that arranges to keep a declining number of backups with age. The following pruning
parameters are supported:

decay-start
The age after backups become eligible for pruning, in days. Only backups more than
this many days old will be pruned. The default is 1 and the minimum is 1.

decay-limit
The age after which backups are always pruned, in days. Backups older than this
will always be pruned unless this would leave no backups at all. The default is
366 and the minimum is 1.

decay-scale
The scale at which the decay window is expanded. The default is 2 and the minimum
is 2.

decay-window
The size of the decay window. The default is 1 and the minimum is 1.

exec
This policy executes a subprogram with parameters and additional information supplied in
the environment.

The following parameters are supported:

path The path to the subprogram to execute.

Any additional parameters are supplied to the subprogram via environment variables,
prefixed with PRUNE_. Additionally the following environment variables are set:

PRUNE_DEVICE
The name of the device containing the backup.

PRUNE_HOST
The name of the host.

PRUNE_ONDEVICE
The list of backups on the device, by age in days. This list excludes any that
have already been scheduled for pruning, and includes the backup under
consideration (i.e. the value of BACKUP_AGE will appear in this list).

PRUNE_TOTAL
The total number of backups of this volume on any device. Note that it does not
include backups on other devices that have just been selected for pruning by
another call to the subprogram.

PRUNE_VOLUME
The name of the volume.

These environment variables all override any parameters with clashing names.

The output should be a list of backups to prune, one per line (in any order). Each line
should contain the age in days of the backup to prune (i.e. the same value as appeared in
PRUNE_ONDEVICE), followed by a colon, followed by the reason that this backup is to be
pruned.

As a convenience, if the argument to prune-policy starts with / then the exec policy is
chosen with the policy name as the path parameter.

never
This policy never deletes any backups.

HOOKS

A hook is a command executed by rsbackup just before or just after some action. The
command is passed directly to execvp(3); to use a shell command, therefore, either wrap it
in a script or invoke the shell with the -c option.

All hooks are run in --dry-run mode. Hook scripts must honor RSBACKUP_ACT which will be
set to false in this mode and true otherwise.

Access Hooks
Access hooks are executed (once) before doing anything that will access backup devices
(even just to read them).

The following environment variables are set when an access hook is executed:

RSBACKUP_ACT
Set to false in --dry-run mode and true otherwise.

RSBACKUP_DEVICES
A space-separated list of known device names.

RSBACKUP_HOOK
The name of the hook (i.e. pre-access-hook, etc). This allows a single hook script
to serve as the implementation for multiple hooks.

Backup Hooks
Backup hooks are executed just before or just after a backup is made.

The following environment variables are set when a backup hook is executed:

RSBACKUP_ACT
Set to false in --dry-run mode and true otherwise.

RSBACKUP_DEVICE
The target device name for the backup.

Note that this may be removed in a future version.

RSBACKUP_HOOK
The name of the hook (i.e. pre-backup-hook, etc). This allows a single hook script
to serve as the implementation for multiple hooks.

RSBACKUP_HOST
The name of the host.

RSBACKUP_SSH_HOSTNAME
The SSH hostname of the host.

Recall that rsbackup treats the hostname localhost specially. If the hook also
needs to do so then it must duplicate this logic.

RSBACKUP_SSH_TARGET
The SSH hostname and username combined for passing to ssh(1).

This will be username@hostname or just hostname depending on whether a SSH username
was set.

RSBACKUP_SSH_USERNAME
The SSH username of the host. If no SSH username was set, this variable will not
be set.

RSBACKUP_STATUS
(Only for post-backup-hook). Either ok or failed.

RSBACKUP_STORE
The path to the store directory where the device is mounted.

RSBACKUP_VOLUME
The name of the volume.

RSBACKUP_VOLUME_PATH
The path to the volume.

The error output from backup hooks is stored in the same backup record as the output from
rsync.

NOTE: The current behavior is that the pre/post backup hooks are run separately for each
backup. In a future version, they may be run only once for all backups of a given volume,
in which case RSBACKUP_DEVICE will no longer be set.

See rsbackup-snapshot-hook(1) for a hook program that can be used to back up from Linux
LVM snapshots.

BACKUP LIFECYCLE

Adding A New Host
To add a new host create a host entry for it in the configuration file.

To back up the local host, specify hostname localhost. Otherwise you can usually omit
hostname.

You may want to set host-wide values for prune-age, max-age and min-backups.

A host with no volumes has no effect.

Adding A New Volume
To add a new volume create a volume entry for it in the relevant host section of the
configuration file.

Add exclude options to skip files you don't want to back up. This might include temporary
files and the contents of "trash" directories.

If the volume contains mount points, and you want to back up the contents of the
subsiduary filesystems, then be sure to include the traverse option.

You may want to set per-volume values for prune-age, max-age and min-backups.

Adding A New Device
To add a new device, format and mount it and create a device-id file in its top-level
directory. Add a device entry for it in the configuration file and a store entry
mentioning its usual mount point.

Under normal circumstances you should make sure that the backup filesystem is owned by
root and mode 0700.

Making Backups
To backup up all available volumes to all available devices:

rsbackup --backup

You will probably want to automate this. To only back up a limited set of volumes specify
selection arguments on the command line.

Pruning Backups
To prune old backups:

rsbackup --prune --prune-incomplete

You will probably want to automate this.

An "incomplete backup" occurs when a backup of a volume fails or is interrupted before
completion. They are not immediately deleted because rsync may be able to use the files
already transferred to save effort on subsequent backups on the same day, or (if there are
no complete backups to use for this purpose) later days.

Retiring A Host
Retiring a host means removing all backups for it. The suggested approach is to remove
configuration for it and then use rsbackup --retire HOST to remove its backups too. You
can do this the other way around but you will be prompted to check you really meant to
remove backups for a host still listed in the configuration file.

If any of the backups for the host are on a retired device you should retire that device
first.

Retiring A Volume
Retiring a volume means removing all backups for it. It is almost the same as retiring a
whole host but the command is rsbackup --retire HOST:VOLUME.

You can retire multiple hosts and volumes in a single command.

Retiring A Device
Retiring a device just means removing the records for it. Use rsbackup --retire-device
DEVICE to do this. The contents of the device are not modified; if you want that you must
do it manually.

You can retire multiple devices in a single command.

RESTORING

Restore costs extra l-)

Manual Restore
The backup has the same layout, permissions etc as the original system, so it's perfectly
possible to simply copy files from a backup directory to their proper location.

Be careful to get file ownership right. The backup is stored with the same numeric user
and group ID as the original system used.

Until a backup is completed, or while one is being pruned, a corresponding .incomplete
file will exist. Check for such a file before restoring any given backup.

Restoring With rsync
Supposing that host chymax has a volume called users in which user home directories are
backed up, and user rjk wants their entire home directory to be restored, an example
restore command might be:

rsync -aSHz --numeric-ids /store/chymax/users/2010-04-01/rjk/. chymax:~rjk/.

You could add the --delete option if you wanted to restore to exactly the status quo ante,
or at the opposite extreme --existing if you only wanted to restore files that had been
deleted.

You might prefer to rsync back into a staging area and then pick files out manually.

Restoring with tar
You could tar up a backup directory (or a subset of it) and then untar it on the target.
Remember to use the --numeric-owner option to tar.

STORE VALIDITY

A store may be in the following states:

available
The store can be used for a backup.

unavailable
The store cannot be used for a backup. Normally this does not generate an error
but --warn-store can be used to report warnings for all unavailable stores, and if
no store is available then the problems with the unavailable stores are described.

bad The store cannot be used for a backup. This always generates an error message, but
does not prevent backups to other stores taking place.

fatally broken
The store cannot be used for a backup. The program will be terminated.

The states are recognized using the following tests (in this order):

· If the store path does not exist, the store is bad.

· If the store does not have a device-id file then it is unavailable. If it has one
but reading it raises an error then it is bad.

· If the store's device-id file contains an unknown device name then it is bad.

· If the store's device-id file names the same device as some other store then it is
fatally broken.

· If the store is not owned by root then it is bad. This check can be overridden
with the public directive.

· If the store can be read or written by group or world then it is bad. This check
can be overridden with the public directive.

Use rsbackup online using onworks.net services