ncftpspooler - Online in the Cloud

PROGRAM:

NAME

ncftpspooler - Global batch FTP job processor daemon

SYNOPSIS

ncftpspooler -d [options]

ncftpspooler -l [options]

OPTIONS

Command line flags:
-d Begin background processing of FTP jobs in the designated FTP job queue directory.

-q XX Use this option to specify a directory to use as the FTP job queue instead of the
default directory, /var/spool/ncftp.

-o XX Use this option to specify a filename to use as the log file. By default, (and
rather inappropriately) the program simply uses a file called log in the job queue
directory. If you don't want a log, use this option to specify /dev/null.

-l Lists the contents of the job queue directory.

-s XX When the job queue is empty, the program sleeps 120 seconds and then checks again
to see if a new job has been submitted. Use this option to change the number of
seconds used for this delay.

DESCRIPTION

The ncftpspooler program evolved from the ncftpbatch program. The ncftpbatch program was
originally designed as a ``personal FTP spooler'' which would process a single background
job a particular user and exit when it finished; the ncftpspooler program is a ``global
FTP spooler'' which stays running and processes background jobs as they are submitted.

The job queue directory is monitored for specially-named and formatted text files. Each
file serves as a single FTP job. The name of the job file contains the type of FTP job
(get or put), a timestamp indicating the earliest the job should be processed, and
optionally some additional information to make it easier to create unique job files (i.e.
a sequence number). The contents of the job files have information such as the remote
server machine to FTP to, username, password, remote pathname, etc.

Your job queue directory must be readable and writable by the user that you plan to run
ncftpspooler as, so that jobs can be removed or renamed within the queue.

More importantly, the user that is running the program will need adequate privileges to
access the local files that are involved in the FTPing. I.e., if your spooler is going to
be processing jobs which upload files to remote servers, then the user will need read
permission on the local files that will be uploaded (and directory access permission the
parent directories). Likewise, if your spooler is going to be processing jobs which
download files, then the user would need to be able to write to the local directories.

Once you have created your spool directory with appropriate permissions and ownerships,
you can run ncftpspooler -d to launch the spooler daemon. You can run additional spoolers
if you want to process more than FTP job from the same job queue directory simultaneously.
You can then monitor the log file (i.e., using tail -f ) to track the progress of the
spooler. Most of the time it won't be doing anything, unless job files have appeared in
the job queue directory.

JOB FILE NAMES

When the ncftpspooler program monitors the job queue directory, it ignores any files that
do not follow the naming convention for job files. The job files must be prefixed in the
format of X-YYYYMMDD-hhmmss where X denotes a job type, YYYY is the four-digit year, MM is
the two-digit month number, DD is the two-digit day of the month, hh is the two-digit hour
of the day (00-23), mm is the two-digit minute, and ss is the two-digit second. The date
and time represent the earliest time you want the job to be run.

The job type can be g for a get (download from remote host), or p for aput (upload to
remote host).

As an example, if you wanted to schedule an upload to occur at 11:45 PM on December 7,
2001, a job file could be named

p-20011207-234500

In practice, the job files include additional information such as a sequence number or
process ID. This makes it easier to create unique job file names. Here is the same
example, with a process ID and a sequence number:

p-20011207-234500-1234-2

When submitting job files to the queue directory, be sure to use a dash character after
the hhmmss field if you choose to append any additional data to the job file name.

JOB FILE CONTENTS

Job files are ordinary text files, so that they can be created by hand. Each line of the
file is a key-pair in the format variable=value, or is a comment line beginning with an
octothorpe character (#), or is a blank line. Here is an example job file:

# This is a NcFTP spool file entry.
job-name=g-20011016-100656-008299-1
op=get
hostname=ftp.freebsd.org
xtype=I
passive=1
remote-dir=pub/FreeBSD
local-dir=/tmp
remote-file=README.TXT
local-file=readme.txt

Job files are flexible since they follow an easy-to-use format and do not have many
requirements, but there are a few mandatory parameters that must appear for the spooler to
be able to process the job.

op The operation (job type) to perform. Valid values are get and put.

hostname
The remote host to FTP to. This may be an IP address or a DNS name (i.e.
ftp.example.com).

For a regular get job, these parameters are required:

remote-file
The pathname of the file to download from the remote server.

local-file
The pathname to use on the local server for the downloaded file.

For a regular put job, these parameters are required:

local-file
The pathname of the file to upload to the remote server.

remote-file
The pathname to use on the remote server for the uploaded file.

For a recursive get job, these parameters are required:

remote-file
The pathname of the file or directory to download from the remote server.

local-dir
The directory pathname to use on the local server to contain the downloaded items.

For a recursive put job, these parameters are required:

local-file
The pathname of the file or directory to upload to the remote server.

remote-dir
The directory pathname to use on the remote server to contain the uploaded items.

The rest of the parameters are optional. The spooler will attempt to use reasonable
defaults for these parameters if necessary.

user The username to use to login to the remote server. Defaults to ``anonymous'' for
guest access.

pass The password to use in conjunction with the username to login to the remote
server.

acct The account to use in conjunction with the username to login to the remote server.
The need to specify this parameter is extremely rare.

port The port number to use in conjunction with the remote hostname to connect to the
remote server. Defaults to the standard FTP port number, 21.

host-ip The IP address to use in conjunction with the remote hostname to connect to the
remote server. This parameter can be used in place of the hostname parameter, but
one or the other must be used. This parameter is commonly included along with the
hostname parameter as supplemental information.

xtype The transfer type to use. Defaults to binary transfer type (TYPE I). Valid
values are I for binary, A for ASCII text.

passive Whether to use FTP passive data connections (PASV) or FTP active data connections
(PORT). Valid values are 0 for active, 1 for passive, or 2 to try passive, then
fallback to active. The default is 2.

recursive
This can be used to transfer entire directory trees. By default, only a single
file is transferred. Valid values are yes or no.

delete This can be used to delete the source file on the source machine after
successfully transferring the file to the destination machine. By default, source
files are not deleted. Valid values are yes or no.

job-name
This isn't used by the program, but can be used by an entity which is
automatically generating job files. As an example, when using the -bbb flag with
ncftpput, it creates a job file on stdout with a job-name parameter so you can
easily copy the file to the job queue directory with the suggested job name as the
job file name.

pre-ftp-command

post-ftp-command
These parameters correspond to the -W, and -Y options of ncftpget and ncftpput.
It is important to note that these refer to RFC959 File Transfer Protocol commands
and not shell commands, nor commands used from within /usr/bin/ftp or ncftp.

pre-shell-command

post-shell-command
These parameters provide hooks so you can run a custom program when an item is
processed by the spooler. Valid values are pathnames to scripts or executable
programs. Note that the value must not contain any command-line arguments -- if
you want to do that, create a shell script and have it run your program with the
command-line arguments it requires.

Generally speaking, post-shell-command is much more useful than pre-shell-command since if
you need to use these options you're more likely to want to do something after the FTP
transfer has completed rather than before. For example, you might want to run a shell
script which pages an administrator to notify her that her 37 gigabyte file download has
completed.

When your custom program is run, it receives on standard input the contents of the job
file (i.e. several lines of variable=value key-pairs), as well as additional data the
spooler may provide, such as a result key-pair with a textual description of the job's
completion status.

post-shell-command update a log file named /var/log/ncftp_spooler.

#!/usr/bin/perl -w

my ($line);
my (%params) = ();

while (defined($line = <STDIN>)) {
$params{$1} = $2
if ($line =~ /^([^=\#\s]+)=(.*)/);
}

if ((defined($params{"result"})) &&
($params{"result"} =~ /^Succeeded/))
{
open(LOG, ">> /var/log/ncftp_spooler.log")
or exit(1);
print LOG "DOWNLOAD" if ($params{"op"} eq "get");
print LOG "UPLOAD" if ($params{"op"} eq "put");
print LOG " ", $params{"local-file"}, "\n";
close(LOG);
}

DIAGNOSTICS

The log file should be examined to determine if any ncftpspooler processes are actively
working on jobs. The log contains copious amounts of useful information, including the
entire FTP control connection conversation between the FTP client and server.

Use ncftpspooler online using onworks.net services