=========================================================
How to use the fonctional test infrastructure for Nupyf ?
=========================================================

1. Overview
===========

There is two scripts:

``run_test.sh``
    This script processes the test. See ``run_test.sh -h`` for more information.
``add_test.sh``
    This script adds or update a testcase. See ``add_test.sh -h`` for more information.
``remode_tests.sh``
    This script pull testcases from Subversion. The URL by default checks out a private repository internal to INL. But you can use this script for your own remote testcases.

Directories contained in ``tests/`` are called "testcases".

Each testcase contains a same ``desc.xml`` file, a same ``nupyf.cmd`` and some testname directories.

A testname directory contains an ACL file called ``acls.xml``, as saved by Nuface. 

A testname directory also contains an expect directory called ``expects``, containing the result files expected for a non-regressing nupyf execution. It is important to keep these files up to date if the nupyf output is modified.

A special file is contained in ``expects``, called ``ldap.ldif``, contains an modified export of the LDAP database. This file is automatically created when using ``add_test.sh`` with ``-l`` option.

2. Installation
===============

Setting variables are defined at the top of scripts. You should take an eyes into them in order to verify if the default value are compliant to your system.

 * ``NUPYF=/usr/sbin/nupyf``: path to nupyf command ;
 * ``OUTPUT_DIR=/tmp/nupyf-tests``: path to temporary path where generating output files ;
 * ``IPTABLES=/sbin/iptables``: path to iptables command ;
 * ``LDAP_HOST=localhost``, ``LDAP_USER="cn=admin,dc=test,dc=lld"``, ``LDAP_PASS=secret``, ``LDAP_BASE="ou=acls,dc=test,dc=lld"``: settings for nupyf to connect to the LDAP server ;

3. Usage
========

3.1 Running tests
-----------------

A/ Classic tests
~~~~~~~~~~~~~~~~

To run all testcases in the same time, just launch the script without argument::

  ./run_script.sh

To run one particularly testcase, give it as an argument::

  ./run_script.sh standard

B/ Automatic tests
~~~~~~~~~~~~~~~~~~

Normally, the script stops when an error occurs.
In order to set an automatic process, it would be useful to process all tests even if some errors occur (for example, to receive a report by e-mail).

To do that, use the -c option::

  ./run_script.sh -c

C/ LDAP tests
~~~~~~~~~~~~~

For testing nupyf with NuFW support, you should set up an LDAP server. If done, you can run tests like this::

  ./run_script.sh -l

:Nota bene: you must be root to run test with LDAP option, because of reinitialization of slapd before each test.

3.2 Adding a new test
---------------------

If needed, create a new testcase (by creating a subdirectory in ``tests/``) and copy the appropriate ``desc.xml`` and ``nupuf.cmd`` into it.

Next, there are two ways to add an ACL file and its expected results.

A/ Taking results from an specific directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you have just called nupyf (via Nuface web interface, for instance) and you would want to integrate the generated result as a test, then call the following command::

  ./add_test.sh -i /etc/network/firewall/dyn/nufw -a /var/local/nuface/acls.xml my_testcase testname

The script will create ``tests/my_testcase/testname/``, copy the ``acls.xml`` in it, and ``nufw/`` contents in ``tests/my_testcase/testname/expects/``.

:Nota Bene: Note that result files usually contains a timestamp on the first line of the file which will be considered as differences (so errors) while running tests. You must remove this heading line before adding the test. ``run_script.sh`` will automatically remove this line for the file it generates himself. To do that, just launch: ``for RULES in /etc/network/firewall/dyn/nufw/*; do tail -n +2 $RULES > /an/other/dir/`basename $RULES`; done``

B/ Generating results with run_test.sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you have not the expected result files (because you have handly edited your own ACL file, you can just copy the ACL file to the testcase (be carefull to the filename: ``acls.xml``) and call ``run_test.sh`` with the -d option not to delete generated results after errors (NB: incompatible with -c option)::

  cp /any/directory/acls.xml my_testcase/example/acls.xml
  ./run_script.sh -d my_testcase

The script will tell that there is no ``my_testcase/example/expects`` directory. Then you can call ``add_test.sh`` without -i and -a options. Like it, the script know it just have to use the already-defined ``acls.xml`` file directly and the result files from the temporary directory::

  ./add_script.sh my_testcase example

:Nota bene: if your ACLs are intended to be used with NuFW for authenticating connection, you should dump the LDAP contents generated by nupyf. To do this, you can use the ``-l`` option of the ``add_test.sh`` script, with any previously described method. For B/, do not forget to also run ``run_test.sh`` with ``-l`` option.

3.3 Updating a test
-------------------

Take care to not update a regression. Read carefully the results you want to replace. ``run_script.sh`` helps you by giving you the diffs between changing files.

To update a test (generally, because the output have changed), you can use the previous A/ or B/ way described in section 2.3. But to indicate you really want tu overwrite the existing results files (and/or ACL file), you have to use the -f option::

  A/
  ./add_test.sh -f -i /etc/network/firewall/dyn/nufw -a /var/local/nuface/acls.xml my_testcase testname

  B/
  ./run_script.sh -d my_testcase
  ./add_script.sh -f my_testcase example

3.4 Removing a test
-------------------

To remove a test, you just have to delete the testname directory.

To remove an entire testcase, you just have to delete the testcase's directory.

:Nota Bene: remember to use ``svn rm`` if the subdirectory are subversioned.

3. Credits
==========

Copyright (c) 2007 INL.

Programmed and documented by Damien Boucard <damien.boucard AT inl.fr>.

Scripts are licensed under the GNU General Public License, version 2.

Documentation is licensed under the GNU Free Documentation License.

For more information, please contact the author or <nuface AT inl.fr>.

