yarn - Online in the Cloud

PROGRAM:

NAME

yarn - scenario testing of Unix command line tools

SYNOPSIS

yarn [--allow-missing-steps] [--no-allow-missing-steps] [--cd-datadir] [--no-cd-datadir]
[--config=FILE] [--dump-config] [--dump-setting-names] [--generate-manpage=TEMPLATE] [-h]
[--help] [--help-all] [--list-config-files] [--version] [--no-default-configs]
[--dump-memory-profile=METHOD] [--env=NAME=VALUE] [--log=FILE] [--log-keep=N]
[--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE] [--memory-dump-interval=SECONDS]
[--output=FILE] [-q] [--quiet] [--no-quiet] [--require-assumptions]
[--no-require-assumptions] [-rSCENARIO] [--run=SCENARIO] [--shell=SHELL] [--shell-arg=ARG]
[-sSHELL-LIBRARY] [--shell-library=SHELL-LIBRARY] [--snapshot] [--no-snapshot]
[--tempdir=DIR] [--timings] [--no-timings] [-v] [--verbose] [--no-verbose] [-n] [--no-act]
[--dry-run] [--pretend] [--no-no-act] [--no-dry-run] [--no-pretend] [FILE]...

DESCRIPTION

yarn is a scenario testing tool: you write a scenario describing how a user uses your
software and what should happen, and express, using very lightweight syntax, the scenario
in such a way that it can be tested automatically. The scenario has a simple, but strict
structure:

GIVEN some setup for the test
WHEN thing that is to be tested happens
THEN the post-conditions must be true

As an example, consider a very short test scenario for verifying that a backup program
works, at least for one simple case.

SCENARIO backups can be restored
GIVEN some live data in a directory
AND an empty backup repository
WHEN a backup is made
THEN the data case be restored
FINALLY cleanup

Note the addition of AND: you can have multiple GIVEN, WHEN, and THEN statements. The AND
keyword makes the text be more readable. SCENARIO is also necessary, and gives the title.

FINALLY is for cleanups. The FINALLY steps will be run regardless of whether the scenario
succeeds or not.

Scenarios are meant to be written in somewhat human readable language. However, they are
not free form text. In addition to the GIVEN/WHEN/THEN structure, the text for each of
the steps needs a computer-executable implementation. This is done by using IMPLEMENTS.
The backup scenario from above might be implemented as follows:

IMPLEMENTS GIVEN some live data in a directory
rm -rf "$TESTDIR/data"
mkdir "$TESTDIR/data"
echo foo > "$TESTDIR/data/foo"

IMPLEMENTS GIVEN an empty backup repository
rm -rf "$TESTDIR/repo"
mkdir "$TESTDIR/repo"

IMPLEMENTS WHEN a backup is made
backup-program -r "$TESTDIR/repo" "$TESTDIR/data"

IMPLEMENTS THEN the data can be restored
mkdir "$TESTDIR/restored"
restore-program -r "$TESTDIR/repo" "$TESTDIR/restored"
diff -rq "$TESTDIR/data" "$TESTDIR/restored"

IMPLEMENTS FINALLY cleanup
echo nothing to do, actually

Each "IMPLEMENTS GIVEN" (or WHEN, THEN, FINALLY) is followed by a regular expression on
the same line, and then a shell script that gets executed to implement any step that
matches the regular expression. The implementation can extract data from the match as
well: for example, the regular expression might allow a file size to be specified.

The above example is a bit silly, of course: why go to the effort to obfuscate the various
steps? The answer is that the various steps, implemented using IMPLEMENTS, can be
combined in many ways, to test different aspects of the program being tested.

Moreover, by making the step descriptions be human language text, matched by regular
expressions, most of the test can hopefully be written, and understood, by non-
programmers. Someone who understands what a program should do, could write tests to
verify its behaviour. The implementations of the various steps need to be implemented by
a programmer, but given a well-designed set of steps, with enough flexibility in their
implementation, that quite a good test suite can be written.

The shell commands in an IMPLEMENTS section are run in the directory in which the user ran
yarn. The environment variable SRCDIR is set to the fully qualified path to that
directory.

OPTIONS

--allow-missing-steps
allow scenarios to reference steps that do not exist, by warning about them, but
otherwise ignoring the scenarios

--no-allow-missing-steps
opposite of --allow-missing-steps

--cd-datadir
change to DATADIR when running commands

--no-cd-datadir
opposite of --cd-datadir

--env=NAME=VALUE
add NAME=VALUE to the environment when tests are run

--generate-manpage=TEMPLATE
SUPPRESSHELP

-h, --help
show this help message and exit

--output=FILE
write output to FILE, instead of standard output

-q, --quiet
be quiet, avoid progress reporting, only show errors

--no-quiet
opposite of --quiet

--require-assumptions
require ASSUMING to always pass

--no-require-assumptions
opposite of --require-assumptions

-r, --run=SCENARIO
run only SCENARIO (this option can be repeated)

--shell=SHELL
run IMPLEMENTS using SHELL

--shell-arg=ARG
use ARG when running shell

-s, --shell-library=SHELL-LIBRARY
include a shell library for the IMPLEMENTS sections to use

--snapshot
make snapshots of test working directory after each scenario step; you probably
want to use this with --tempdir

--no-snapshot
opposite of --snapshot

--tempdir=DIR
use DIR as the temporary directory for tests; it should be empty or not exist

--timings
report wall clock time for each scenario and step

--no-timings
opposite of --timings

-v, --verbose
make progress reporting be more verbose ("wall of text"), instead of a one-line
status info; this is turned automatically if there is not terminal

--no-verbose
opposite of --verbose

--version
show program's version number and exit

-n, --no-act, --dry-run, --pretend
do not actually run any tests, merely print what would be run

--no-no-act, --no-dry-run, --no-pretend
opposite of --no-act

Configuration files and settings
--config=FILE
add FILE to config files

--dump-config
write out the entire current configuration

--dump-setting-names
SUPPRESSHELP

--help-all
show all options

--list-config-files
SUPPRESSHELP

--no-default-configs
clear list of configuration files to read

Logging
--log=FILE
write log entries to FILE (default is to not write log files at all); use "syslog"
to log to system log, "stderr" to log to the standard error output, or "none" to
disable logging

--log-keep=N
keep last N logs (10)

--log-level=LEVEL
log at LEVEL, one of debug, info, warning, error, critical, fatal (default: debug)

--log-max=SIZE
rotate logs larger than SIZE, zero for never (default: 0)

--log-mode=MODE
set permissions of new log files to MODE (octal; default 0600)

Peformance
--dump-memory-profile=METHOD
make memory profiling dumps using METHOD, which is one of: none, simple, or meliae
(default: simple)

--memory-dump-interval=SECONDS
make memory profiling dumps at least SECONDS apart

ENVIRONMENT

DATADIR
Fully qualified pathname to a temporary directory, in which the tests can use
files. The temporary directory is removed at the end of the test execution, unless
the user specifies otherwise with --snapshot.

SRCDIR Fully qualitifed pathname to the directory in which the user ran yarn. This is
useful when the tests want to change the directory.

EXAMPLE

To run yarn on all the scenarios in your current directory:

yarn *.scenario

All the files will be treated together as if they had been one file.

To add a shell library to be included when running any IMPLEMENTS section:

yarn --shell-library mylib.sh *.scenario

You can repeat --shell-library as many times as necessary.

Use yarn online using onworks.net services