xjobs - Online in the Cloud

PROGRAM:

NAME

xjobs - construct command line and execute jobs in parallel

SYNOPSIS

xjobs [options] [utility [argument ...]]

DESCRIPTION

xjobs reads job descriptions line by line and executes them in parallel. It limits the
number of parallel executing jobs and starts new jobs when jobs finish. Therefore, it
combines the arguments from every input line with the utility and arguments given on the
command line. If no utility is given as an argument to xjobs, then the first argument on
every job line will be used as utility. To execute utility xjobs searches the directories
given in the PATH environment variable and uses the first file found in these directories.

xjobs is most useful on multiprocessor machines when one needs to execute several time
consuming commands that could possibly be run in parallel. With xjobs this can be achieved
easily, and it is possible to limit the load of the machine to a useful value. It works
similar to xargs, but starts several processes simultaneously and gives only one line of
arguments to each utility call.

By using I/O redirectors the standard input, output, and error stream of executed jobs can
be redirected. Use < to redirect standard input, > to redirect standard output, >! to
redirect standard output and overwrite an existing file, >> to append standard output to
an existing file, >& to redirect both standard output and standard error output to the
same file, and >>& to append both standard output and standard error output to the same
file.

If passed on the command line, these operators specify the default I/O redirection that
can be overwritten by specifying another redirector to a specific job on its argument
line. After all these operators a filename is expected. See EXAMPLES below for an
example. If you need more advanced shell features than the redirection operators
supported by xjobs, then use as utility a shell of your preference.

Every job line can be preceeded by a "cd directory;" command that tells xjobs in which
directory the job shall be executed. For every line this can only be used once. For more
complex scripting, please pass the line to execute to a shell of your choice.

xjobs constructs the arguments of the jobs to execute from each input line. Each input
line will create a seperate job, whereas newline character are handled as regular
whitespace by xargs. To be able to include whitespace charakters in arguments, either
preceed them with a backslash or quote them with single or doublequote charakters. A
backslash charakter preceeding a newline will make xjobs ignore the newline character,
thus giving you the ability to pass arguments for a single job across multiple lines. To
include quotation marks in quoted arguments, preceed them with a backslash. Lines passed
to xjobs beginning with a # charakter are interpreted as comments.

Finally, xjobs also includes a mechanism for serializing the execution. Like this it is
possible to parallelize independent jobs and sequence jobs that have a dependency. This
can be achieved by inserting a line that only consists of two percentage charakters in
sequence (%%). All jobs before this sequence point are executed at the requested number
of jobs in parallel. When hitting the sequence point xjobs waits for all processes to
finish and then continues starting jobs that follow the sequence point.

When passing a named pipe (i.e. a file name created by mkfifo) via option -s as an input,
xjobs will close and reopen the fifo when reaching end-of-file. Like this it is possible
to setup an xjobs server and sending jobs to this server from muliple programs. See
section EXAMPLES below for an example.

OPTIONS

-j <jobs>
Sets the maximum number of jobs that are started in parallel. The default value is
to limit the number executing jobs is equal to the number of online processors in
the system. If the number passed as <jobs> is followed by an 'x' charakter (e.g.
2.5x), the value is multiplied with the number of online processors before setting
the job limit. I.e. having a machine with 4 online processors and passing 2.5x as
an argument to option -j will yield a joblimit of 10 jobs.

-s <script>
Use file script instead of the standard input to read the job descriptions.

-n Redirect standard output and standard error output of executed jobs to /dev/null.

-l <num>
Combine the arguments of <num> input lines for a single job.

-p Start jobs interactively, prompting the user.

-q <num>
Limits the number of queued jobs to num elements. Normally xjobs reads in jobs from
standard input or the give script and queues them if they cannot be started at
once. With this option, xjobs will stop reading as soon as num jobs are queued and
restart reading when a new job has been started. Like this xjobs allocates less
memory. Use this option, if you pass huge number of jobs to xjobs, to limit memory
consumption. It can also increase performance of xjobs, but be sure that jobs get
fed fast enough to xjobs.

-1 Pass one argument per job, which is expected to be terminated by a new-line
character. No argument parsing is performed. That way it is more easy to process
jobs where arguments may include whitespace character or other tokens that
influence argument parsing.

-0 Same as -1, but as a job and argument termination character a null-character (\0)
is expected instead of a new-line character. That way also arguments with new-line
character can be processed without escape sequences.

-V Print the version number of xjobs and exit.

-v <level>
Set verbosity of xjobs to level. Valid leves are: 0=silent, 1=error, 2=warning,
3=info, 4=debug. The default level of verbosity is 3.

EXAMPLES

If you have a lot of .zip files that you want to extract, then use xjobs like this:

$ ls -1 *.zip | xjobs unzip

If you want to do the same without getting the output of each unzip task on your terminal,
then try this:

$ ls -1 *.zip | xjobs -n unzip

To gzip all *.bak files in a given directory hierarchy, use it the following way:

$ find . -name '*.bak' | xjobs gzip

To generate index files for a set of *.jar files, you can use the redirection feature of
xjobs, and do the following:

$ ls -1 *.jar | sed 's/\(.*\)/\1 > \1.idx/' | xjobs jar tf

If you also want to capture the error output, than use >& instead of >.

You can also use it to execute several different commands. Therefore, write a script file
that contains every job you want to execute and pass it to xjobs with the option -s:

$ cat - > script
unzip my.zip
tar xf my.tar
lame --silent my.wav my.mp3
crypt notsecret < mydata > secretfile
^D
$ xjobs -s script

To be able to queue up jobs from multiple sources with xjobs, use a named pipe and pass it
explicitly as input script. Then write the jobs to the named pipe:
$ mkfifo /var/run/my_named_pipe
$ xjobs -s /var/run/my_named_pipe &
$ echo unzip 1.zip >> /var/run/my_named_pipe
$ echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe

ENVIRONMENT VARIABLES

PATH Determines the location of command.

AUTHORS

Thomas Maier-Komor <[email protected]>
Donations via PayPal are welcome!

HOMEPAGE

http://www.maier-komor.de/xjobs.html

Use xjobs online using onworks.net services