gfpcopy - Online in the Cloud

PROGRAM:

NAME

gfpcopy - copy Gfarm files in parallel

SYNOPSIS

gfpcopy [-nqvdpPU] [-X regexp] [-S source-domainname] [-h source-hostfile]
[-D destination-domainname] [-H destination-hostfile] [-j num-of-processes]
[-J num-of-processes] [-M limit-byte] [-z minimum-byte] [-Z maximum-byte]
[-w way-of-scheduling] [-W kilobytes-for-threshold] [-s KB/s-to-simulate]
[-F num-for-readahead] [-b bufsize] [-f] [-e] [-k] source-path destination-path

DESCRIPTION

gfpcopy copies files in parallel.

When the source-path parameter is a directory, files under the directory will be copied
recursively.

When the destination-path does not exist, the directory is created. When the
destination-path exists, a directry of the same name as the source-path is created under
the destination-path.

A set of source/destination hosts can be specified by a domain name and/or a hostlist
file. When both a domain name and a hostlist file are specified, a set of hosts is
determined by both conditions. When a set of source hosts is specified, only files stored
on the source hosts are cppied. When a set of source hosts are specified, only files
stored on the set of source hosts are copied. A set of hosts can be specified by a domain
name and/or a host file. When both a domain name and a host file are specified, a set of
hosts is determined by both conditions.

gfpcopy can also read the directory and get information of the entries in parallel.

SOURCE PATH

source-path must be following format. Files on HPSS cannot be copied.

gfarm:...
is Gfarm URL.

file:...
is URL of Local Filesystem.

ANY
is relative path or absolute path of Local Filesystem. When the path is a mountpoint
on gfarm2fs, files are copied without passing through the gfarm2fs.

DESTINATION PATH

destination-path must be following format.

gfarm:...
is Gfarm URL of a directory.

file:...
is URL of a directory on Local Filesystem.

hpss:...
is URL of a directory on HPSS. If A directory named the same as the source-path exists
under this directory, the source-path cannot be copied. The difference of files cannot
be copied to HPSS. Relative path such as "hpss:". or "hpss:dir" can be also specified.

ANY
is relative path or absolute path of Local Filesystem. When the path is a mountpoint
on gfarm2fs, files are copied without passing through the gfarm2fs.

GFPCOPY OPTIONS

These are the options only for gfpcopy.

-b bufsize
Specifies the buffer size to copy.

The default value is 64KiB (64 * 1024).

-f
With -f option, existing destination files will be overwritten when the size is
different or the modification time (mtime) is different from the source file.

Without -f option, existing destination files will be overwritten when that they are
older than the corresponding source files in the modification time.

-e
Skips existing destination files in order to execute multiple gfpcopy simultaneously.

-k
Does not copy symbolic links.

COMMON OPTIONS

gfpcopy has the same options with gfprep. (see man gfprep)

EXAMPLES

To copy files under the directory recursively.

$ gfpcopy gfarm:///dir file:///tmp/dir

$ gfpcopy file:///tmp/dir gfarm:///dir

$ gfpcopy gfarm:///dir1 gfarm:///dir2

$ gfpcopy gfarm:///dir hpss:///tmp/dir

$ cd /mnt/gfarm2fs
$ gfpcopy dir /tmp/dir

To copy a file.

$ gfpcopy gfarm:///dir/file file:///dir

$ cd /mnt/gfarm2fs
$ gfpcopy file /tmp/dir

NOTES

If the gfmd and gfpcopy are far apart, the gfpcopy operation is slow. Specifying a large
value by -j or -J options may be effective in such a case.

Use gfpcopy online using onworks.net services