pegasus-kickstart - Online in the Cloud

PROGRAM:

NAME

pegasus-kickstart - remote job wrapper

SYNOPSIS

pegasus-kickstart [-n tr] [-N dv] [-H] [-R site] [-W | -w dir]
[-L lbl -T iso] [-s p | @fn] [-S p | @fn] [-i fn]
[-o fn] [-e fn] [-X] [-l fn sz] [-F] (-I fn | app [appflags])
pegasus-kickstart -V

DESCRIPTION

pegasus-kickstart is a wrapper program which manages and monitors the execution of jobs on
remote resources.

Sitting in between the remote scheduler and the application process, it is possible for
pegasus-kickstart to gather additional information about the process' run-time behavior
and resource usage, including the exit status of jobs. This information is important for
Pegasus invocation tracking as well as detecting Globus GRAM job failures.

pegasus-kickstart allows the optional execution of jobs before and after the main
application job that run in chained execution with the main application job. See section
SUBJOBS for details about this feature.

It also allows stdin, stdout, and stderr to be redirected from/to specific files.

All jobs with relative path specifications to the application are part of search relative
to the current working directory (yes, this is unsafe), and by prepending each component
from the PATH environment variable. The first match is used. Jobs that use absolute
pathnames, starting in a slash, are exempt. Using an absolute path to your executable is
the safe and recommended option.

pegasus-kickstart rewrites the command line of any job (pre, post and main) with variable
substitutions from Unix environment variables. See section VARIABLE REWRITING below for
details on this feature.

OPTIONS

-n tr
In order to associate the minimal performance information of the job with the
invocation records, the jobs needs to carry which transformation was responsible for
producing it. The format is the textual notation for fully-qualified definition names,
like namespace::name:version, with only the name portion being mandatory.

There is no default. If no value is given, "null" will be reported.

-N dv
The jobs may carry which instantiation of a transformation was responsible for
producing it. The format is the textual notation for fully-qualified definition names,
like namespace::name:version, with only the name portion being mandatory.

There is no default. If no value is given, "null" will be reported.

-H
This option avoids pegasus-kickstart writing the XML preamble (entity), if you need to
combine multiple pegasus-kickstart records into one document.

Additionally, if specified, the environment and the resource usage segments will not
be written, assuming that a in a concatenated record version, the initial run will
have captured those settings.

-R site
In order to provide the greater picture, pegasus-kickstart can reflect the site handle
(resource identifier) into its output.

There is no default. If no value is given, the attribute will not be generated.

-L lbl, -T iso
These optional arguments denote the workflow label (from DAX) and the workflow’s last
modification time (from DAX). The label lbl can be any sensible string of up to 32
characters, but should use C identifier characters. The timestamp iso must be an ISO
8601 compliant time-stamp.

-S l=p
If stat information on any file is required before any jobs were started, logical to
physical file mappings to stat can be passed using the -S option. The LFN and PFN are
concatenated by an equals (=) sign. The LFN is optional: If no equals sign is found,
the argument is taken as sole PFN specification without LFN.

This option may be specified multiple times. To reduce and overcome command line
length limits, if the argument is prefixed with an at (@) sign, the argument is taken
to be a textual file of LFN to PFN mappings. The optionality mentioned above applies.
Each line inside the file argument is the name of a file to stat. Comments (#) and
empty lines are permitted.

Each PFN will incur a statcall record (element) with attribute id set to value
initial. The optional lfn attribute is set to the LFN stat’ed. The filename is part of
the statinfo record inside.

There is no default.

-s fn
If stat information on any file is required after all jobs have finished, logical to
physical file mappings to stat can be passed using the -s option. The LFN and PFN are
concatenated by an equals (=) sign. The LFN is optional: If no equals sign is found,
the argument is taken as sole PFN specification without LFN.

This option may be specified multiple times. To reduce and overcome commandline length
limits, if the argument is prefixed with an at (@) sign, the argument is taken to be a
textual file of LFN to PFN mappings. The optionality mentioned above applies. Each
line inside the file argument is the name of a file to stat. Comments (#) and empty
lines are permitted.

Each PFN will incur a statcall record (element) with attribute id set to value final.
The optional lfn attribute is set to the LFN stat’ed. The filename is part of the
statinfo record inside.

There is no default.

-i fn
This option allows pegasus-kickstart to re-connect the stdin of the application that
it starts. Use a single hyphen to share stdin with the one provided to
pegasus-kickstart.

The default is to connect stdin to /dev/null.

-o fn
This option allows pegasus-kickstart to re-connect the stdout of the application that
it starts. The mode is used whenever an application produces meaningful results on its
stdout that need to be tracked by Pegasus. The real stdout of Globus jobs is staged
via GASS (GT2) or RFT (GT4). The real stdout is used to propagate the invocation
record back to the submit site. Use the single hyphen to share the application’s
stdout with the one that is provided to pegasus-kickstart. In that case, the output
from pegasus-kickstart will interleave with application output. For this reason, such
a mode is not recommended.

In order to provide an un-captured stdout as part of the results, it is the default to
connect the stdout of the application to a temporary file. The content of this
temporary file will be transferred as payload data in the pegasus-kickstart results.
The content size is subject to payload limits, see the -B option. If the content grows
large, only the last portion will become part of the payload. If the temporary file
grows too large, it may flood the worker node’s temporary space. The temporary file
will be deleted after pegasus-kickstart finishes.

If the filename is prefixed with an exclamation point, the file will be opened in
append mode instead of overwrite mode. Note that you may need to escape the
exclamation point from the shell.

The default is to connect stdout to a temporary file.

-e fn
This option allows pegasus-kickstart to re-connect the stderr of the application that
it starts. This option is used whenever an application produces meaningful results on
stderr that needs tracking by Pegasus. The real stderr of Globus jobs is staged via
GASS (GT2) or RFT (GT4). It is used to propagate abnormal behavior from both,
pegasus-kickstart and the application that it starts, though its main use is to
propagate application dependent data and heartbeats. Use a single hyphen to share
stderr with the stderr that is provided to pegasus-kickstart. This is the backward
compatible behavior.

In order to provide an un-captured stderr as part of the results, by default the
stderr of the application will be connected to a temporary file. Its content is
transferred as payload data in the pegasus-kickstart results. If too large, only the
last portion will become part of the payload. If the temporary file grows too large,
it may flood the worker node’s temporary space. The temporary file will be deleted
after pegasus-kickstart finishes.

If the filename is prefixed with an exclamation point, the file will be opened in
append mode instead of overwrite mode. Note that you may need to escape the
exclamation point from the shell.

The default is to connect stderr to a temporary file.

-l logfn
allows to append the performance data to the specified file. Thus, multiple XML
documents may end up in the same file, including their XML preamble. stdout is
normally used to stream back the results. Usually, this is a GASS-staged stream. Use a
single hyphen to generate the output on the stdout that was provided to
pegasus-kickstart, the default behavior.

Default is to append the invocation record onto the provided stdout.

-w dir
permits the explicit setting of a new working directory once pegasus-kickstart is
started. This is useful in a remote scheduling environment, when the chosen working
directory is not visible on the job submitting host. If the directory does not exist,
pegasus-kickstart will fail. This option is mutually exclusive with the -W dir option.

Default is to use the working directory that the application was started in. This is
usually set up by a remote scheduling environment.

-W dir
permits the explicit creation and setting of a new working directory once
pegasus-kickstart is started. This is useful in a remote scheduling environment, when
the chosen working directory is not visible on the job submitting host. If the
directory does not exist, pegasus-kickstart will attempt to create it, and then change
into it. Both, creation and directory change may still fail. This option is mutually
exclusive with the -w dir option.

Default is to use the working directory that the application was started in. This is
usually set up by a remote scheduling environment.

-X
make an application executable, no matter what. It is a work-around code for a
weakness of globus-url-copy which does not copy the permissions of the source to the
destination. Thus, if an executable is staged-in using GridFTP, it will have the wrong
permissions. Specifying the -X flag will attempt to change the mode to include the
necessary x (and r) bits to make the application executable.

Default is not to change the mode of the application. Note that this feature can be
misused by hackers, as it is attempted to call chmod on whatever path is specified.

-B sz
Changes the amount of stdout and stderr data to include in the output. The last sz
bytes of the stdout and stderr of the process will be copied into kickstart’s output.
All other data will be discarded. The special value all can be used to capture all the
stdout/stderr of the process. The default is 256KB.

-F
This flag will issue an explicit fsync() call on kickstart’s own stdout file.
Typically you won’t need this flag. Albeit, certain shared file system situations may
improve when adding a flush after the written invocation record.

The default is to just use kickstart’s NFS alleviation strategy by locking and
unlocking stdout.

-I fn
In this mode, the application name and any arguments to the application are specified
inside of file fn. The file contains one argument per line. Escaping from Globus,
Condor and shell meta characters is not required. This mode permits to use the maximum
possible command line length of the underlying operating system, e.g. 128k for Linux.
Using the -I mode stops any further command line processing of pegasus-kickstart
command lines.

Default is to use the app flags mode, where the application is specified explicitly on
the command-line.

-f
This flag causes kickstart to output full information, including the environment and
resource limits under which the job ran, and any useful auxilliary statcalls. If the
job fails, then -f is implied.

-t
This flag causes kickstart to skip tracing the child process and omit the <proc>
element. This flag only exists when kickstart is compiled for Linux.

-q
This flag causes kickstart to omit the <data> part of the <statcall> records when the
job exits successfully. This is designed to reduce the size of the output logs for
large workflows.

app
The path to the application has to be completely specified. The application is a
mandatory option.

appflags
Application may or may not have additional flags.

RETURN VALUE

pegasus-kickstart will return the return value of the main job. In addition, the error
code 127 signals that the call to exec failed, and 126 that reconnecting the stdio failed.
A job failing with the same exit codes is indistinguishable from pegasus-kickstart
failures.

Use pegasus-kickstart online using onworks.net services