pegasus-exitcode - Online in the Cloud

PROGRAM:

NAME

pegasus-exitcode - Checks the stdout/stderr files of a workflow job for any indication
that an error occurred in the job. This script is intended to be invoked automatically by
DAGMan as the POST script of a job.

SYNOPSIS

pegasus-exitcode [-h][-r rv][-n][-s msg][-f msg] job.out

DESCRIPTION

pegasus-exitcode is a utility that examines the STDOUT of a job to determine if the job
failed, and renames the STDOUT and STDERR files of a job to preserve them in case the job
is retried.

Pegasus uses pegasus-exitcode as the DAGMan postscript for jobs submitted via Globus GRAM.
This tool exists as a workaround to a known problem with Globus and Condor-G where the
exitcodes of GRAM jobs are not returned. This is a problem because Pegasus uses the
exitcode of a job to determine if the job failed or not.

In order to get around the exitcode problem, Pegasus can wrap GRAM jobs with Kickstart,
which records the exitcode of the job in an XML invocation record, which it writes to the
job’s STDOUT. The STDOUT is transferred from the execution host back to the submit host
when the job terminates. After the job terminates, DAGMan runs the job’s postscript, which
Pegasus sets to be pegasus-exitcode. pegasus-exitcode looks at the invocation record
generated by kickstart to see if the job succeeded or failed. If the invocation record
indicates a failure, then pegasus-exitcode returns a non-zero result, which indicates to
DAGMan that the job has failed. If the invocation record indicates that the job succeeded,
then pegasus-exitcode returns 0, which tells DAGMan that the job succeeded.

In addition, clustered jobs executed with pegasus-cluster or pegasus-mpi-cluster will have
a [cluster-summary] record in their STDOUT. pegasus-exitcode can examine these records to
determine if any of the tasks in the clustered job failed.

pegasus-exitcode performs several checks (some optional) to determine whether a job failed
or not. These checks include:

1. Is the Condor exitcode non-zero? If so, then the job failed.

2. Is STDOUT empty? If it is empty, then the job failed.

3. Are there any failure messages in the STDOUT or STDERR? If so, the job failed.

4. Are all of the success messages in the STDOUT or STDERR? If not, then the job failed.

5. Does the [cluster-summary] record indicate that the job was successful. If not, then
the job failed.

6. Are there any <status> tags with a non-zero value? If there are, then the job failed.
Note that, if this is a clustered job, there could be multiple <status> tags, one for
each task. If any of them are non-zero, then the job failed.

7. Is there at least one <status> tag with a zero value? There must be at least one
successful invocation or the job has failed.

In addition, pegasus-exitcode allows the caller to specify the exitcode returned by Condor
using the --return argument. This can be passed to pegasus-exitcode in a DAGMan post
script by using the $RETURN variable. If this value is non-zero, then pegasus-exitcode
returns a non-zero result before performing any other checks. For GRAM jobs, the value of
$RETURN will always be 0 regardless of whether the job failed or not.

In addition to checking the success/failure of a job, pegasus-exitcode also renames the
STDOUT and STDERR files of the job so that if the job is retried, the STDOUT and STDERR of
the previous run are not lost. It does this by appending a sequence number to the end of
the files. For example, if the STDOUT file is called "job.out", then the first time the
job is run pegasus-exitcode will rename the file "job.out.000". If the job is run again,
then pegasus-exitcode sees that "job.out.000" already exists and renames the file
"job.out.001". It will continue to rename the file by incrementing the sequence number
every time the job is executed.

OPTIONS

-h, --help
Prints a usage summary with all the available command-line options.

-r rv, --return rv
Return value reported by DAGMan. This can be specified in the DAG using the $RETURN
variable. If this is non-zero, then pegasus-exitcode immediately fails with a non-zero
return value itself. If it is zero, then just rotate the file and don’t check for
proper kickstart output. This option can be used in cases where kickstart cannot be
used (such as pegasus-create-dir) to enable file rotation.

-n, --no-rename
Don’t rename job.out and job.err to .out.XXX and .err.XXX. This option is used
primarily for testing.

-f msg, --failure-message msg
Failure message to find in job stdout/stderr. If this message exists in the
stdout/stderr of the job, then the job will be considered a failure no matter what
other output exists. If multiple failure messages are provided, then none of them can
exist in the output or the job is considered a failure.

-s msg, --success-message msg
Success message to find in job stdout/stderr. If this message does not exist in the
stdout/stderr of the job, then the job will be considered a failure no matter what
other output exists. If multiple success messages are provided, then they must all
exist in the output or the job is considered a failure.

AUTHORS

Gideon Juve <juve@usc.edu>

Pegasus Team http://pegasus.isi.edu

Use pegasus-exitcode online using onworks.net services