Introduction
============

Atheist is a simple framework for running test cases. You write small
files in Python language using a set of predefined functions and
classes. In many senses, the concept is quite similar to *make* or the
SCons framework although Atheist is not a building system at all.

Features:

* `Black-box testing
  <http://en.wikipedia.org/wiki/Black-box_testing>`_: system and
  acceptance tests. It may execute any kind of external shell program.
* `White-box testing
  <http://en.wikipedia.org/wiki/White-box_testing>`_ by means of
  `unitest  <http://docs.python.org/library/unittest.html>`_  and standard
  Python functions.
* `Python doctests  <http://en.wikipedia.org/wiki/Doctest>`_.
* `Python unittest <http://pyunit.sourceforge.net/pyunit.html>`_.
* New kinds of "tests" by means of plugins.
* Plugable pre- and post- condition system.


ToDo:

* Per-project plugins.
* Limit/perf testing.
* Remote testing.
* Test deployment.
* Remote management.
* Distributed testing.


Test objects
------------

The Test object is the minimal testing unit. Each Test instance
defines an individual execution (a shell command) that may be
checked for success upon termination. The Test constructor accepts many
parameters that may change the test's exception behavior in several
ways. The only mandatory parameter is ``cmd`` which is the command to
execute.

The Test object is responsible for executing the command and checking its
termination value. A very basic example::

   Test('true')


Test files (TestCases)
----------------------

Test instances need to be defined in text source files (with
``.test`` extension). Although these files are written in a subset of
the Python language, they may be seen as declarative programs. You tell
Atheist what you want to test and even the order, but the decision
about when to run the corresponding action is taken by Atheist; some
of them may be never done.

The file **does not define sequential** imperative sentences. For example,
if you write this in a ``.test`` file::

   Test('ls /')
   print "hello"

the ``print`` statement will be executed when the test is **LOADED**
and the ``ls`` command will run later, when the test is
**EXECUTED**. You must take in mind that the atheist .test file defines
a set of tests. It is not a conventional python program.


Key-args
^^^^^^^^

Any Test constructor accepts the next key-val parameters. All of them
are optional. Beside the parameter's name appear its type and
default value.

**check** -- type: ``bool``, default: ``True``

   If 'check' is ``False`` there is no consequences when the Task
   fails and it is not considered in the final stats.

**cwd** -- type: ``str``

   Directory for the task execution.

**delay** -- type: ``int``, default: ``0``

   Waits for 'delay' seconds before launching the task actions.

**desc** -- type: ``str``

   One-liner textual task description.

**detach** -- type: ``bool``, default: ``False``

   When ``detach`` is ``False`` the next task does not start until
   the current one ends. When ``detach`` is ``True`` the next task is
   executed even if the current one is still running.

**env** -- type: ``str``:``str`` map

   A dictionary with shell environment variables.

**expected** -- type: int

   Expected return code for the command. It may be negative if the process is
   killed by a signal.

**tid** -- type: ``str``

   It is a unique string Task IDentifier. You can get a reference to
   the task later by giving this value to the :func:`get_task` function.

**must_fail** -- type: ``bool``, default: ``False``

   When you expect the program to end with an error but the
   return code is not known (i.e: is different from zero). You should
   check other conditions (stdout, stderr, generated files, etc) to differentiate
   alternative fails.

.. _compositetask:

**parent** -- type: CompositeTask

   Use this to aggregate the current test to an already defined CompositeTask.

**template** -- type: Template list

   A list of templates. See :ref:`templates`.

**timeout** -- type: int, default: ``5``

   The maximum task execution time (in seconds). When the timeout
   finishes, Atheist sends the programmed signal to the process. To
   avoid timed termination (daemonic task) give ``timeout=0``.

**save_stderr** -- type: ``bool``, default: ``False``

   Store the process' stderr in a text file. If the ``stderr``
   parameter is not set, Atheist will create an unique name for the file.

**save_stdout** -- type: ``bool``, default: ``False``

   Store the process' stdout in a text file. If the ``stdout``
   parameter is not set, Atheist will create an unique name for the file.

**shell** -- type: ``bool``, default: ``False``

   Execute the command within a shell session. ``bash`` is the used shell.

**signal** -- type: int, default: SIGKILL

   It is the signal that Atheist sends to the process when the
   ``timeout`` finishes.

**stdout** -- type: ``str``

   It is the name of the file where to save the process' stdout. Setting this
   parameters implies save_stdout = True.

**todo** -- type: ``bool``, default: ``False``

   It indicates that the task is not fully verified and it is possible that it
   fail unduly. This has no effect when the task ends
   successfully.

Not all of these key-args are available for all Task classes. See :ref:`task_types`.


Task results
------------

The execution of any task returns a value, which can be:

* **FAIL**: The task ran normally but user requirements or conditions were not met. The test **failed**.
* **OK**: The task ran successfully and all required conditions and/or return values were correct.
* **NOEXEC**: The task was skipped or it was not executed.
* **ERROR**: The implementation of the task is wrong and the task execution failed itself.
* **UNKNOWN**: The task was executed but its result is not known.
* **TODO**: The task implementation is unstable and it may produce false failures.


.. _templates:

Templates
---------

The template is a set of predefined values for Test
key-values. You may use the same configuration for many tests avoiding
to repeat them. This is an example::


    t1 = Template(timeout=6, expected=-9)
    Test('foo', templates=[t1])
    Test('bar', templates=[t1])


Both tests will be automatically killed after 6 seconds and the
expected return value is -9. This means that these processes receive the
SIGKILL(9) signal. You may specify several templates as a list.



Conditions
----------

*Conditions* are predicates (actually, functors) that check for
specific conditions. Conditions may be specified to be checked before
(pre-conditions) or after (post-conditions) the task execution. If
any of the conditions fail, then the task fails. This is an example::

  t = Test('foo')
  t.pre  += FileExists('/path/to/foofile')
  t.post += FileContains('path/to/barfile', 'some text')


Available builtin conditions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. function:: AtheistVersion(version)

   Checks that the installed version of **atheist** is equal or newer than the
   given number. This is useful to assure recent or experimental features.

.. function:: Callback(*args)

   Call the specified function with the given arguments. You must
   avoid the use of this condition as much as possible.

.. function:: DirExists(path)

   Checks that directory *path* exists.

.. function:: EnvVarDefined(name[, value])

   Checks that the environment variable *name* exists, and optionally has
   the value *value*.

.. function:: FileExists(filename)

   Checks that file *filename* exists.

.. function:: FileContains(val[, filename=task.stdout, strip='', whole=False, times=1])

   Checks that file *filename* exists and contains *val*, which may be a
   string or a string list.

   If *whole* is ``False`` (default) val must be a strip and the whole
   content of file must be equal to *val*.

   Otherwise, the file must contain **at least** *times*
   occurrences of *val*. The default value for *filename* is the stdout of
   the corresponding task, and it implies ``save_stdout=True``. This
   implies also the automatic creation of a FileExists condition for
   that file.

   The file content may be stripped by means of *strip* argument. No
   stripping by default.


.. function:: FileEquals(filename1[, filename2=task.stdout])

   Checks that the contents of *filename1* and *filename2* are
   identical. The default value for *filename2* is the stdout of the
   current task, and it implies ``save_stdout=True``.

.. function:: OpenPort(port[, host='localhost'[, proto='tcp']])

   Checks that port number *port* is open, i.e., a process is listening to it.

.. function:: ProcessRunning(pid)

   Checks that the given PID belongs to a running process.

There are other available conditions as plugins.


Condition decorators
^^^^^^^^^^^^^^^^^^^^

.. function:: Not(condition)

   It is True when *condition* is evaluated as False.

   Example::

     t = Test('rm foo_file')
     t.pre += Not(FileExists(foo_file))


.. function:: Poll(condition[, interval=1[, timeout=5]])

   Checks *condition* every *interval* seconds, stopping after *timeout*
   seconds or when its value becomes True.

   In the next example, the task waits (a maximum of 5 seconds) for the ``nc`` server to become
   ready before continuing::

     t = Test('nc -l -p 2000')
     t.post += Poll(OpenPort(2000))


Condition decorators may be combined. The following example shows a task that waits
for an environment variable to be removed before executing the command::

   t = Test('command')
   t.pre += Poll(Not(EnvVarDefined('WAIT_FLAG')), timeout=10)

Note that the effect of Poll(Not(condition)) is not the same that Not(Poll(condition)).


.. _task_types:


Task's, Test's, Commands, Daemons...
------------------------------------

``Task`` is the base class for all executable items. ``Test`` is-a
``Task`` that runs a shell command but other kind of ``Task`` are
possible:

**Command**
  It is a non-checked ``Test``. Command is exactly the same that a Test
  with a ``check=False`` parameter.

  The Commands (or other non-checked Tasks) are not considered in
  results counting.

**Daemon**
  Command shortcut for detached commands. Predefined parameters are:

  * ``detach = True``
  * ``expected = -9`` (sigkilled)
  * ``timeout = None`` (runs "forerver")
  * ``check = False``

**TestFunc**
  Check the return value of arbitrary Python functions or methods. A return
  value of 0 means success, otherwise is a error code. For unit
  testing, prefer `unittestcase`_ instead of TestFunc.


There are other available Task subclasses as plugins.


Function utilities
------------------

.. function:: get_task(name)

   Returns the task whose ``tid`` attribute is *name*.
   ToDo: [include a sample here]

.. function:: load(filename)

   Makes possible to reuse atheist or python code in other
   files. ``load()`` returns a module-like object that may be used to
   access to functions, classes and variables defined in the "loaded"
   module. All atheist classes are available within the loaded modules::

    common = load("files/common.py")

    Test("./server %s" % common.server_params())

.. warning:: Previous ``include()`` function is not supported any more.


Variable substitutions
----------------------

Test files may include some substitutable variable. This is useful
to locate test-relevant related files. You must write the
symbol '$' preceding each one of the next words:

**basedir**
  is the name of the directory where atheist was
  executed. Usually this is a ``tests`` directory into your project.

**dirname**
  is the name of the directory where the testfile is.

**fname**
  is the path to the testfile without its extension (.test).

**testname**
  is just the name of the testfile, without extension nor directory
  path.

For example, for the following ``vars.test`` file::

     Test("echo $basedir $dirname $fname $testname")

When you run atheist, you get::

     ~/sample$ atheist -i2 tests/vars.test
     [ OK ] Test case: ./tests/substitutions.test
     [ OK ] +- T-1   ( 0: 0)  echo . ./tests ./tests/vars vars
     Total:  ALL OK!!  - 0.24s - 1 test


hooks: setup and teardown
-------------------------

You may write tasks to execute before and after **each** test file. To do
so, just put this tasks in files called ``_setup.test`` and
``_teardown.test``.


Clean-up
--------

When your task creates files you may track them for automatic
cleaning. Just add their filenames to the task ``gen`` attribute. Here's an
example::

    t = Test('touch foo')
    t.gen += 'foo'

You may specify one o more filenames (as a string list).

If you want the generated files not to be automatically removed for
manual inspection of the results, you must specify the ``--dirty`` option
(see below). To clean-up these files later, specify the ``-C`` option.


Invoking Atheist
================

.. program:: atheist

.. cmdoption:: -h, --help

   Show basic help information.

.. cmdoption:: -a ARGS, --task-args=ARGS

   Colon-separated options for the tests.

.. cmdoption:: -b PATH, --base-dir=PATH

   Change working directory.

.. cmdoption:: -C, --clean-only

   Don't execute anything, only remove generated files.

.. cmdoption:: -d, --describe

   Describe tasks but don't execute anything.

.. cmdoption:: -e, --stderr

   Print the test process' stderr.

.. cmdoption:: -f, --out-on-fail

   Print task output, but only if it fails.

.. cmdoption:: -g, --gen-template

   Generate a test file template with default values.

.. cmdoption:: -i LEVEL, --report-detail=LEVEL

   Verbosity level (0:nothing, [1:case], 2:task, 3:composite, 4:condition)

.. cmdoption:: -j, --skip-hooks

   Skip _setup and _teardown files.

.. cmdoption:: -k, --keep-going

   Continue despite failed tests.

.. cmdoption:: -l, --list

   List tests but do not execute them.

.. cmdoption:: -o, --stdout

   Print the test process' stdout.

.. cmdoption:: -p PATH, --plugin-dir=PATH

   Load plugins from that directory (may be given several times).

   Print the test process' stdout.

.. cmdoption:: -q, --quiet

   Do not show result summary nor warnings, only totals.

.. cmdoption:: -r RANDOM, --random=RANDOM

   Run testcases in random order using the specified seed.

.. cmdoption:: -s INLINE, --script=INLINE

   Specifies command line script.

.. cmdoption:: -t, --time-tag

   Include time info in the logs.

.. cmdoption:: -u, --until-failure

   Repeat tests until something fails.

.. cmdoption:: -v, --verbose

   Increase verbosity.

.. cmdoption:: -w WORKERS, --workers=WORKERS

   Number of simultaneous tasks. '0' allows atheist to choose the number. Default is 1.

.. cmdoption:: -x COMMAND, --exec=COMMAND

   Exec COMMAND like in "Test(COMMAND, shell=True)"

.. cmdoption:: --case-time

   Print case execution time in reports

.. cmdoption:: --cols=WIDTH

   Set terminal width (in chars).

.. cmdoption:: --config=FILE

   Alternate config file.

.. cmdoption:: --dirty

   Don't remove generated files after test execution.

.. cmdoption:: --disable-bar

   Don't show progress bar.

.. cmdoption:: --ignore=PATTERN

   Files to ignore (glob patterns) separated with semicolon.

.. cmdoption:: --log=LOG

   List tasks but do not execute them.

.. cmdoption:: --plain

   Avoid color codes in console output.

.. cmdoption:: --save-stdout

   Save stdout of all tasks

.. cmdoption:: --save-stderr

   Save stderr of all tasks

.. cmdoption:: --version

   Atheist version

.. cmdoption:: --notify-jabber=JABBER

   Notify failed tests to the given jabber account (may be given several times).

.. cmdoption:: --notify-smtp=MAIL

   Notify failed tests to the given email address (may be given several times).



Logging control
---------------

**[ToDo]**


Result report
-------------

**[ToDo]**


Config files
============

**[ToDo]**


Notifications (other reporters)
===============================

You may use Atheist to monitor the proper working of any application. It
may send you notifications when something is wrong. The easiest way is to
run a testcase with ``cron`` specifying one ``--notify-*`` command-line
argument. Currently, two notificators are implemented:

**Email**
  The destination is a email account using the SMTP
  protocol. Atheist does not require an SMTP server or smarthost. You
  only need to configure an email account that will be used by Atheist
  to send mail to any destination. That information must be written in
  the ``~/.atheist/config`` configuration file. This is an example::

    [smtp]
    host = smtp.server.org
    port = 587
    user = atheist.notify@server.org
    pasw = somesecret


**Jabber**
  You must indicate a jabber account that will be used by
  Atheist to send notification messages to any other jabber
  account. The destination account must accept this contact
  previously. The following is an example for the configuration in the
  ``~/.atheist/config`` file::

    [jabber]
    user = atheist.notify@server.org
    pasw = anothersecret

To ask for a notification you just need to specify a test file and
one or more destinations. For example::

    $ atheist --notify-jabber John.Doe@jabber.info test/some_test.test

It is possible to give several --notify-* arguments in the same
command-line to send notifications to different destinations.



.. Local variables:
..   coding: utf-8
..   ispell-local-dictionary: "american"
.. End:
