path: root/doc/devel/atf.dox

/**
@page tests Testing

@section testsOverview Testing Overview

In DHCP, a unit test exercises a particular piece of code in
isolation. There is a separate unit test per module or API. Each unit
test lives in a directory beneath the code it is designed to exercise.
So, we (will eventually) have:

@verbatim
server/tests/
client/tests/
common/tests/
dhcpctl/tests/
...
@endverbatim

And so on.

Ideally each function would be invoked with every possible type of input, and
each branch of every function would be checked. In practice we try to be a bit
more pragmatic, and target the most basic operations, as well tricky code, and
areas we have seen bugs in the past.

We are using <a href="http://code.google.com/p/kyua/wiki/ATF">ATF (Automated
Test Framework)</a> as a framework to run our unittests.

@section testsAtf ATF unit-tests

ATF stands for Automated Test Framework, and is the framework used for unit
tests in ISC DHCP.  Since ATF was created, a next generation QA toolchain
called, Kyua, has beend developed that can execute ATF unit tests.  There are
two approaches one can take to install the necessary software for executing
ISC DHCP unit tests:

1) Install pre-Kyua release (0.19 or earlier) of ATF

Using this approach is the simplest as it involves installing only ATF. However
it requires using a version of ATF that is no longer maintained and which may
eventually may be unavailable.

ATF may be found here: https://github.com/jmmv/atf


2) Install post-Kyua release of ATF (0.20 or later) and Kyua

Using this approach requires the installation of more software as Kyua has
dependencies that ATF lacks.  The advantage is that the version(s) that can
be used are all still supported.

Kyua may be found here: https://github.com/jmmv/kyua

@subsection preKyuaATF Using pre-Kyua ATF

This section describes how to install and use pre-Kyua ATF release to run
ISC DHCP unit tests.  The most recent release of ATF that can be used without
also needing Kyua, is ATF version 0.19.  While this version is still available
and using it is least software to install to run ISC DHCP unit tests, there
may come a point in time when this version is no longer available. It may be
downloaded here: https://github.com/jmmv/atf/releases/tag/atf-0.19

After downloading and unpacking the source, you'll need to configure
it, build it, and then install it.  Note you'll need to include the
"--enable-tools" option when configuring ATF.   The following would build
and install it to /opt/atf/atf-0.19/local:

@verbatim

$ cd /opt/atf
$ tar -xvf $HOME/Downloads/atf-0.19.tar.gz

# While the --prefix is optional, you must specify --enable-tools
# as a configure option

$ ./configure --prefix=/opt/atf/atf-0.19/local --enable-tools
$ make
$ sudo make install

@endverbatim

To build and run the ISC DHCP unit-tests, use the following:

@verbatim
$ git clone https://gitlab.isc.org/isc-projects/dhcp.git
$ cd dhcp

# Note because we used --prefix when configuraing ATF above, we
# need to tell ISC DHCP configure where it is located.

$ ./configure --with-atf=/opt/atf/atf-0.19/local

# Run all of the unit tests via "make check"
$ make check
@endverbatim

This will traverse the source tree running the unit tests in each unit test
subdirectory.  Note that if one or more tests in a unit test subdirectory fail
the make process will stop.  To run all of the tests regardless of outcome,
use "make -k check"

Note, If you installed ATF to its default location rather than using --prefix
or you install the pkg-config .pc files for ATF, you should be able to
simply specify "--with-atf" without a path when configuring ISC DHCP.

Each dhcp source sub-directory (e.g. server, common) that has unit-tests has a
sub-directory named tests (e.g. server/tests). You can execute "make check" in
that sub-directory to run its specific subset of tests.

Unit-tests are grouped into suites, each suite being a separate
executable. The typical way to run tests is:

@verbatim
$ atf-run | atf-report

# (This assumes atf-run and atf-report are in your path)
# or

$ sh ../../tests/unittests.sh
@endverbatim

atf-run will read the Atffile in the current directory and execute all
the tests specified in it. Using atf-run - rather than calling the
test binary directly - has several major benefits. The main one is
that atf-run is able to recover from test segfault and continue
execution from the next case onwards. Another is that it is possible
to specify a timeout for a test. atf-run will kill the test in case of
any infinite loops and will continue running next tests.

It is possible to run atf-run without passing its output to atf-report, but its
output is somewhat convoluted. That is useful in some situations, e.g. when one
wants to see test output.

It is possible to run test binary directly. The only required
parameter is the test case name. The binary will print out a warning
that direct binary execution is not recommended as it won't be able to
recover from crash.  However, such an approach is convenient for
running the test under the debugger.

@subsection KyuaPlusATF Using Kyua with ATF

Kyua is a QA toolchain that has been layered on top of ATF.  The tools
necessary to run ATF unit tests migrated from ATF to within Kyua beginning
with ATF 0.20. This section includes instructions for building and running
ISC DHCP with Kyua. Instructions are included for both Ubuntu 18.04 and MacOS.

@subsubsection KyuaLinux Kyua Ubuntu 18.04 Instructions

The following instructions are based on Ubuntu 18.04, but
are typical of a linux distribution/installation. It installs all software
into default locations for simplicity.  If you want to use
custom installation paths, you may have to set additional environment
variables, library paths, and the like.

Before we install kyua, we must install its dependencies: lua, sqlite3,
ATF, and lutok

@verbatim

# lua and sqlite3 have packages, we'll use them

$ apt-get install liblua5.1-0-dev
$ apt-get install libsqlite3-dev

# Fetch, build, and install the latest version of ATF

$ git clone https://github.com/jmmv/atf.git
$ cd atf
$ autoreconf --install --force
$ ./configure
$ make
$ sudo make install
$ cd ..

# Fetch, build, and install the latest version lutok

$ git clone https://github.com/jmmv/lutok.git
$ cd lutok

# We must use -I argument to tell autoreconf where to find the ATF .m4 files
$ autoreconf --install -s -I/usr/local/share/aclocal

$ ./configure
$ make
$ sudo make install
$ cd ..

# Finally, we fetch, build and install the latest kyua

$ git clone https://github.com/jmmv/kyua.git
$ cd kyua

# We must use -I argument to tell autoreconf where to find the ATF .m4 files
$ autoreconf --install -s -I/usr/local/share/aclocal

$ ./configure
$ make
$ sudo make install
$ cd ..
@endverbatim

To build ISC DHCP with Kyua and run all the unit tests is the same:

@verbatim
$ git clone https://gitlab.isc.org/isc-projects/dhcp.git
$ cd dhcp

# Because we used default installations we simply enable unit tests with
# --with-atf with no path value.

$ ./configure --with-atf
$ make

# Run the ISC DHCP unit tests
$ make check
@endverbatim

@subsubsection KyuaMacOS Kyua macOS Instructions

The following instructions are for based on macOS Mojave (10.1.4.4).
Before we install kyua, we must install its dependencies: pkgconfig, lutok,
and ATF:

@verbatim

# install dependencies
$ brew install pkgconfig
$ brew install lutok

# install atf so we can build kyua and dhcp with unit tests
$ git clone https://github.com/jmmv/atf.git
$ cd atf
$ autoreconf --install --force
$ ./configure --prefix=/opt/atf
$ make
$ sudo make install
$ cd ..

# configure kyua with atf
$ git clone https://github.com/jmmv/kyua.git
$ cd kyua
$ autoreconf --install -s -I/opt/atf/share/aclocal
$ ./configure --prefix=/opt/kyua
$ make
$ sudo make install
$ cd ..
@endverbatim

To build ISC DHCP with Kyua and run all the unit tests is the same:

@verbatim
$ git clone https://gitlab.isc.org/isc-projects/dhcp.git
$ cd dhcp

# Because we used the prefix "/opt/atf" to install ATF we need to
# add that path to --with-atf when we configure ISC DHCP.

$ ./configure --with-atf=/opt/atf
$ make

# Add kyua to the path and then run the ISC DHCP unit tests
# via "make check"
$ export PATH=$PATH:/opt/kyua/bin
$ make check
@endverbatim

@section testsAtfAdding Adding new unit-tests

There are a small number of unit-tests that are not ATF based. They will be
converted to ATF soon. Please do not use any other frameworks.

Sadly, the DHCP code was not written with unit-testing in mind: often a
non-standard approach is required for writing unit-tests. The existing code
often has many dependencies that make testing a single piece of code awkward to
unit test.  For example, to test hash tables, one needs to also include the
OMAPI code. Rather than significantly refactoring the code (a huge task that
could take months), we decided to link whatever is needed in the tests. If
developing new test suite, it is recommended that you take a look at existing
tests and just copy them as a starting point.


In particular, the following
things should be done for adding new tests:

<b>1. Tests directory.</b> For each code component (server, client, common,
etc.) there should be a tests subdirectory. If it isn't there yet, then it must
be created. This can be done by:

a). Creating the directory:

@verbatim
    $ mkdir $subdir/tests
    $ cvs add tests
@endverbatim

b). Adding the subdirectory to the build system:

    Add to $subdir/Makefile.am:

@verbatim
    SUBDIRS = tests
@endverbatim

    Add to the AC_OUTPUT macro in configure.ac:

@verbatim
    subdir/tests/Makefile
@endverbatim

c. Create a Makefile.am in the new directory, something similar to this:

@verbatim
    AM_CPPFLAGS = -I../..

    check_PROGRAMS = test_foo

    TESTS = test_foo

    test_foo_SOURCES = test_foo.c
    test_foo_LDADD = ../../tests/libt_api.a     # plus others...
@endverbatim

See existing Makefile.am for examples, and the Automake documentation:

    http://www.gnu.org/software/automake/manual/html_node/Tests.html

<b>2. Implement the test.</b> That typically means that you create a
new file that will hold test code. It is recommended you name it
(tested_feature_name)_unittest.c and put the file in specified tests
directory.  For example tests related to hash tables used on the
server side should be named server/tests/hash_unittest.c. If in doubt,
it is convenient to name the test code after the file that holds
tested code, e.g. server/mdb6.c is tested in
server/tests/mdb6_unittest.c.

The file server/tests/simple_unittest.c holds a template explaining
the basic layout of the ATF tests.  There may be many test cases in a
single *_unittest.c file. Make sure that you register all your test
cases using ATF_TP_ADD_TC() macro, and try to minimize modifications
to the tested code if possible. Keep in mind that we are using
modernized \ref codingGuidelines for test development. You are advised
to also look at atf-c-api(3) man page.

To add a new test, such as when a new module is added or when you want
to start testing existing code, you can copy the
server/tests/simple_unittest.c as a new new file, add the new file as
a target in Makefile.am, and begin adding tests. Reviewing that file
is a good idea, even if you decide to write your test from scratch, as
it give you quick overview of the essential capabilities of the ATF
framework (how to write test, how to make checks, pass or fail test
etc.). Do not forget to add your new file to git via "git add
yourtest_unittest.c".

<b>3. Extend Makefile.am</b> to build your test. In particular, add your binary
name to ATF_TESTS. The tests directory will be built only in case where
ATF is enabled, using --with-atf during configure phase.

<b>4. Modify Atffile to include your new test</b>, if needed. Tests in
the specified directory must be registered in Atffile. See
server/tests/Atffile for an example. Currently every executable with
name of the form *_unittest will be executed automatically. If you
followed naming convention proposed in a previous step, your test will
be included and will be included automatically.

<b>5. Enjoy your improved confidence in the code</b>, as you can run
the tests after any change you may want to do:

@verbatim
$ make check
@endverbatim

to run all tests for all components. See \ref atfTests section for more details
on running tests.

@section testsAtfCoding ATF Coding Guidelines

As the unit-test code creates an evironment that works under a different
regime than the production code, there are slight differences to standard
coding guidelines. In particular:

- The code is written using C99. Double slash comments are allowed.
- Please do not use tabs. Use 4 spaces for each indent level.

*/