pegasus-transfer - Online in the Cloud

PROGRAM:

NAME

pegasus-transfer - Handles data transfers in Pegasus workflows.

SYNOPSIS

pegasus-transfer [-h]
[--file inputfile]
[--threads number_threads]
[--max-attempts attempts]
[--debug]

DESCRIPTION

pegasus-transfer takes a list of url pairs, either on stdin or with an input file,
determines the correct tool to use for the transfer and executes the transfer. Some of the
protocols pegasus-transfer can handle are GridFTP, SRM, Amazon S3, HTTP, and local
cp/symlinking. Failed transfers are retried.

Note that pegasus-transfer is a tool mostly used internally in Pegasus workflows, but the
tool can be used stand alone as well.

OPTIONS

-h, --help
Prints a usage summary with all the available command-line options.

-f inputfile, --file inputfile
File with input pairs. If not given, stdin will be used.

-m, --max-attempts attempts
Maximum number of attempts for retrying failed transfers.

-t, --threads number_threads
The number of threads to use. This controls the parallelism of transfers.

-d, --debug
Enables debugging output.

EXAMPLE

$ pegasus-transfer
# src 1 local_site
file:///etc/hosts
# dst 1 local_site
file:///tmp/foo
CTRL+D

CREDENTIAL HANDLING

Credentials used for transfers can be specified with a combination of comments in the
input file format and environment variables. For example, give the following input file:

# src 1 isi
gsiftp://workflow.isi.edu/data/file.dat
# dst 1 tacc_stampede
gsiftp://gridftp.stampede.tacc.utexas.edu/scratch/file.dat

pegasus-transfer will expect either one environment variable specifying one credential to
be used on both end of the connection (X509_USER_PROXY), or two separate environment
variables specifying two different credentials to be used on the two ends of the
connection. The the latter case, the environment variables are derived from the site names
provided in the input file commments. In the example above, the environment variables
would be named X509_USER_PROXY_isi and X509_USER_PROXY_tacc_stampede

THREADING

In order to speed up data transfers, pegasus-transfer will start a set of transfers in
parallel using threads. Threads are turned off when retrying failed transfers.

Use pegasus-transfer online using onworks.net services