path: root/doc/qtcreator/src/howto/creator-only/creator-autotest.qdoc

// Copyright (C) 2023 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only

/*!
    \previouspage creator-ctf-visualizer.html
    \page creator-autotest.html
    \nextpage creator-squish.html

    \title Running Autotests

    \QC supports both \e {code based tests} and \e {build system based tests}.
    Code based testing offers special handling for particular testing
    frameworks that strongly ties to the underlying code models or
    specialized parsers. Build system based testing is independent from any
    testing frameworks. It retrieves information directly from the underlying
    build system and uses it or even the build system as such to execute the
    respective tests.

    \QC integrates the following testing frameworks for unit testing
    applications and libraries:

    \list
        \li \l{https://www.boost.org/doc/libs/1_81_0/libs/test/doc/html/index.html}
            {Boost.Test}
        \li \l{https://github.com/catchorg/Catch2}{Catch2 test framework}
        \li \l{https://github.com/google/googletest}{Google C++ Testing Framework}
        \li \l{Qt Test} framework
    \endlist

    In addition, \QC offers build system based support for
    \l{https://cmake.org/cmake/help/latest/manual/ctest.1.html}{CTest}.

    You can use \QC to create, build, and run code based tests for your
    projects.

    \image qtcreator-autotests.png

    \section1 Build System Based Tests

    By default, \QC does not handle build system based tests to avoid
    interference with code based parsers. To enable build system based tests,
    select the respective test tool in \uicontrol Preferences > \uicontrol Testing
    > \uicontrol General.

    The detection of tests is usually much faster for build system based
    tests as this normally does not involve scanning or parsing.
    The information in the tests tree is usually more detailed
    when using code based tests.

    If you enable both code based and build system based tests, you
    may duplicate tests inside the tests tree. See also \l {Selecting Tests to Run}.

    \section1 Creating Tests

    Use a wizard to create projects that have tests.

    \section2 Creating Qt and Qt Quick Tests

    To create a Qt or Qt Quick test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New Project} >
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
            \uicontrol Choose to create a project with boilerplate code for a
            Qt test or a Qt Quick test.

        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:

            \list 1

                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Qt Test} or \uicontrol {Qt Quick Test}.

                \li For a Qt test, select the \uicontrol {GUI Application} check
                    box to create a Qt application.

                    \image qtcreator-autotests-project-qt-test.png {Autotest project wizard - Qt Test}

                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.

                \li For a Qt test, select the \uicontrol {Requires \QApplication}
                    check box to add the include statement for QApplication to
                    the main.cpp file of the project.

                \li For a Qt Quick test, select the
                    \uicontrol {Generate setup code} check box to execute C++
                    code before running any of the QML tests. The testing
                    framework will call slots and invocable functions, as
                    described in \l{Executing C++ Before QML Tests}.

                    \image qtcreator-autotests-project-qtquick-test.png {Autotest project wizard - Qt Quick Test}

                \li Select the \uicontrol {Generate initialization and cleanup
                    code} checkbox to add functions to your test that the
                    testing framework executes to initialize and clean
                    up the test.

                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.

            \endlist

    \endlist

    \QC creates the test in the specified project directory. Edit the .cpp file
    to add private slots for each test function in your test. For more information
    about creating Qt tests, see \l{Creating a Test}.

    \section2 Creating Google Tests

    To create a Google test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New Project} >
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
            \uicontrol Choose to create a project with boilerplate code for a
            Google test.

        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:

            \list 1

                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Google Test}.

                \li In the \uicontrol {Test suite name} field, enter a name for
                    the test suite.

                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.

                \li Select the \uicontrol {Enable C++ 11} check box to
                    support C++ 11 features in the test.

                \li In the \uicontrol {Google test repository} field, select
                    a directory that has a clone of the googletest
                    repository.

                    To use an installed Google C++ Testing framework instead,
                    see \l{Setting Up the Google C++ Testing Framework}.

                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.

            \endlist

    \endlist

    \QC creates the test in the specified project directory.
    For more information about creating Google tests, see the
    \l{https://github.com/google/googletest/blob/master/docs/primer.md}
    {Google Test Primer}.

    \section2 Creating Boost Tests

    To build and run Boost tests, you must have the Boost.Test installed on the
    development host. Typically, the Boost installation includes it. You can
    download Boost from \l{https://www.boost.org/}{Boost.org}.

    If the compiler and build system can find the Boost libraries, you
    do not need to specify the include directory when creating the test.

    To create a Boost test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New Project} >
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
            \uicontrol Choose to create a project with boilerplate code for a
            Boost test.
        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:
            \list 1
                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Boost Test}.
                \li In the \uicontrol {Test suite name} field, enter a name for
                    the test suite.
                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.
                \li In the \uicontrol {Boost include dir (optional)} field,
                    enter the path to the directory that has files needed
                    by Boost.Test, such as \e version.hpp and the \e test
                    subfolder that contains the test header files.
                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.
            \endlist
    \endlist

    \QC creates the test in the project directory.
    For more information about creating Boost tests, see
    \l{https://www.boost.org/doc/libs/1_81_0/libs/test/doc/html/index.html}
    {Boost.Test}.

    \section2 Creating Catch2 Tests

    To build and run Catch2 tests, you can either install Catch2
    libraries and headers or use the single include header file in the
    Catch2 repository.

    If the compiler and build system can find the Catch2 headers automatically,
    you do not need to specify the include directory when creating the test.

    To create a basic Catch2 test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New Project} >
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
            \uicontrol Choose to create a project with boilerplate code for a
            Catch2 test.
        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:
            \list 1
                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Catch2}.
                \li In the \uicontrol {Test case name} field, specify a name
                    for the test case file.
                \li Select the \uicontrol {Use Qt libraries} check box
                    to use a self-defined main function and set up the project
                    to use Qt features.
                \li In the \uicontrol {Catch2 include directory (optional)} field,
                    you may enter a path to the directory that has the
                    Catch2 header files.
                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.
            \endlist
    \endlist

    \QC creates the test in the project directory.
    For more information about creating Catch2 tests, see
    \l{https://github.com/catchorg/Catch2/blob/master/docs/Readme.md}
    {Catch2}.

    \section2 Creating CTest Based Tests

    CTest can execute tests for CMake based projects regardless of the test
    framework. You configure tests in the project file, usually, CMakeLists.txt.
    Basically, you enable testing for the project and register the test
    applications or even special commands.

    \code
    enable_testing()
    add_test(NAME test_example COMMAND test_example)
    \endcode

    Add \c test_example as an executable before trying to register it as test.
    It may be any executable command including arguments.
    For detailed information on how to use CTest see
    \l{https://gitlab.kitware.com/cmake/community/-/wikis/doc/ctest/Testing-With-CTest}
    {Testing with CTest}.

    \section1 Setting Up the Google C++ Testing Framework

    To build and run Google tests, install and configure the Google C++ Testing
    framework on the development host. You can either clone it from Git Hub or
    install it from an installation package.

    To configure a project to use a cloned Google testing framework, edit the
    \c INCLUDEPATH variable in the project file (.pro) to include the source
    and \c include folders of Google Test's \c googletest and \c googlemock.
    Usually, you need to add the following subfolders:

    \list
        \li \c googletest
        \li \c googlemock
        \li \c googletest/include
        \li \c googlemock/include
    \endlist

    Also, add the necessary files to the \c SOURCES variable. For example:

    \list
        \li \c googletest/src/gtest-all.cc
        \li \c googlemock/src/gmock-all.cc
    \endlist

    To configure a project to use an installed Google testing framework package,
    add the following include paths to the .pro file:

    \list
        \li \c <googletest_install_path>/include/gtest
        \li \c <googletest_install_path>/include/gmock
    \endlist

    Then add linker options to be able to find the libraries and to link against
    them. For example, for qmake based projects, you typically need to add the
    following values to the .pro file:

    \list
        \li \c {LIBS += -lgtest -L<path_to_gtest_lib>}
        \li \c {LIBS += -lgmock -L<path_to_gmock_lib>}
    \endlist

    \section1 Building and Running Tests

    To build and run tests:

    \list 1

        \li Open a project that has tests.

        \li In the \uicontrol Tests view, select the tests to run.

        \li In the \uicontrol {Test Results}, select:

            \list
                \li \inlineimage icons/run_small.png
                    (\uicontrol {Run All Tests}) to run all tests.
                 \li \inlineimage icons/qtcreator-run-selected-tests.png
                    (\uicontrol {Run Selected Tests}) to run the selected tests.
                 \li \inlineimage icons/qtcreator-run-failed-tests.png
                    (\uicontrol {Run Failed Tests}) to re-run the tests which failed
                    in the last run.
                    Depending on the framework, this may select more tests if it
                    is impossible to distinguish or to fully address the test.
                 \li \inlineimage icons/qtcreator-run-tests-in-current-file.png
                    (\uicontrol {Run Tests for Current File}) to run the tests
                    in the file currently open in the code editor.
            \endlist

    \endlist

    By default, \QC builds a project before deploying and running it.
    To run all tests without building and deploying them again, select
    \uicontrol {Run All Tests Without Deployment} in the context menu.
    To run the selected tests without deployment, select
    \uicontrol {Run Selected Tests Without Deployment}.

    The functions to run tests are also available in the context menu in the
    \uicontrol Tests view and in \uicontrol Tools > \uicontrol Tests.

            \note If you enable both build system based and code based tests,
            you may run tests twice when using \uicontrol {Run All Tests} or
            \uicontrol {Run Selected Tests}. This happens if the tests can be
            found by the code based test frameworks and are registered as test
            with the build system.

    If a test takes more than a minute to execute, the default timeout might
    stop the test execution. To increase the timeout, select \uicontrol Edit >
    \uicontrol Preferences > \uicontrol {Testing} > \uicontrol General.

    \section2 Selecting Tests to Run

    The \uicontrol Tests view shows all the tests found for the currently active
    test frameworks in the current project. Select the test cases to run.

    \image qtcreator-tests-view.png

    If a Qt Quick test case does not have a name, it is marked
    \uicontrol Unnamed in the list. \uicontrol {Run All Tests} executes
    unnamed test cases. You cannot select or deselect them.

    \QC scans the project for tests when you open the project and updates the
    test list for the currently active test frameworks when you edit tests.
    To refresh the view, select \uicontrol {Rescan Tests} in the context menu.

    To show or hide init and cleanup or data functions in the \uicontrol Tests
    view, select \inlineimage icons/filtericon.png
    (\uicontrol {Filter Test Tree}), and then select \uicontrol {Show Init and
    Cleanup Functions} or \uicontrol {Show Data Functions}. Double-click a
    function in the list to open its source code in the code editor.

    The \uicontrol Tests view lists test cases in alphabetic, case insensitive
    order. To list them in the order in which they appear in the source code,
    select \inlineimage icons/leafsort.png
    (\uicontrol {Sort Naturally}).

    \section2 Running and Debugging Tests from Code Editor

    You can run and debug tests in the file currently open in the code editor.
    To run all tests in the open file, select \uicontrol Tools >
    \uicontrol Tests > \uicontrol {Run Tests for Current File}.
    \note Available only for code based test frameworks.

    To run the test currently selected in the open file, select
    \uicontrol {Run Test Under Cursor} in the context menu.
    \note Available only for code based test frameworks.

    To debug the currently selected test, select
    \uicontrol {Debug Test Under Cursor} in the context menu.
    \note Availability for build system based tests depends on the build system
    tool's abilities.

    \section1 Specifying Test Settings

    To customize the handling of tests, test frameworks, and test tools, select
    \uicontrol Edit > \uicontrol Preferences > \uicontrol {Testing} >
    \uicontrol General.

    \image qtcreator-preferences-testing-general.webp {General tab in Testing preferences}

    You can customize some settings at project level. To change settings
    for the current project instead of globally, select \uicontrol Projects >
    \uicontrol {Project Settings} > \uicontrol {Testing}.

    In the \uicontrol {Active Test Frameworks} list, select tests for \QC to
    handle. To improve the performance of full scans for tests, disable
    test frameworks you do not use.

    To group related test cases for an active test framework, select the
    \uicontrol Group check box next to the framework name in the
    \uicontrol {Active Test Frameworks} list.
    By default, \QC groups tests that are in the same directory.

    \QC omits internal messages and run configuration warnings for
    deduced  configurations by default. To view them, deselect the
    \uicontrol {Omit internal messages} and
    \uicontrol {Omit run configuration warnings} check boxes.

    By default, test result output shows a maximum of 100,000 characters. The
    view automatically scrolls to show the latest results. To display
    full results, deselect the \uicontrol {Limit result output} check box.
    To disable automatic scrolling, deselect the
    \uicontrol {Automatically scroll results} check box.

    Set the maximum number of lines in the test result tooltip and description
    in \uicontrol {Limit result description}.

    To group test results by the executable path that you use to run the
    tests, select \uicontrol {Group results by application}. This is useful
    if you have multiple test executables and run them all at once.

    To automatically run tests after successfully building the current project,
    select them in \uicontrol {Automatically run}.

    Sometimes, \QC cannot deduce which executable or run configuration to use.
    If \QC repeatedly asks you to select the tests to run when trying to execute
    tests, you can let it cache your choices and use them where appropriate. \QC
    clears the cache when you switch to another project, close the current one,
    or select \uicontrol {Reset Cached Choices}.

    Select the \uicontrol {Process arguments} check box to pass arguments to the
    test executable that you specify in the \l {Managing Run Configurations}
    {run configuration}. This is an experimental feature that might cause the
    execution of the test executable to fail.

    \section2 Specifying Settings for Running Qt Tests

    The code inside a benchmark test is measured, and possibly also repeated
    several times in order to get an accurate measurement. This depends on the
    measurement back-end that you can select in the
    \uicontrol {Benchmark Metrics} group in \uicontrol Edit >
    \uicontrol Preferences > \uicontrol {Testing} > \uicontrol {Qt Test}:
    walltime, CPU tick counter, event counter, Valgrind Callgrind, and Linux
    Perf. For more information, see \l{Creating a Benchmark}.

    \image qtcreator-preferences-testing-qttest.webp {Qt Tests tab in Testing preferences}

    To receive verbose output when running benchmarks, select the
    \uicontrol {Verbose benchmarks} check box.

    To allow the debugger to interrupt Qt tests on assertions, select the
    \uicontrol {Disable crash handler while debugging} check box.

    To record information about signals and slots in the test log, select the
    \uicontrol {Log signals and slots} check box.

    To explicitly limit the maximum number of warnings in the test log, select
    the \uicontrol {Limit warnings} check box and set the limit. Set it to 0 if
    you want no limit at all. The default number is 2000.

    To check for Qt Quick Tests that are derived from TestCase, select the
    \uicontrol {Check for derived Qt Quick tests} check box.

    \note This feature is rather expensive and increases the
    scan time significantly.

    \section2 Specifying Settings for Running Google Tests

    To specify settings for running Google tests, select \uicontrol Edit >
    \uicontrol Preferences > \uicontrol {Testing} > \uicontrol {Google Test}.

    \image qtcreator-preferences-testing-googletest.webp {Gooble Test tab in Testing preferences}

    To run disabled tests, select the \uicontrol {Run disabled tests} check box.

    To run several iterations of the tests, select the \uicontrol {Repeat tests}
    check box and enter the number of times to run the tests in the
    \uicontrol Iterations field. To make sure that the tests are independent and
    repeatable, run them in a different order each time by selecting the
    \uicontrol {Shuffle tests} check box. Set the seed for initializing the
    randomizer in the \uicontrol Seed field. The value 0 generates a seed
    based on the current timestamp.

    To turn failures into debugger breakpoints, select the
    \uicontrol {Break on failure while debugging} check box.

    To group Google tests by using a GTest filter, select
    \uicontrol {GTest Filter} in the \uicontrol {Group mode} field,
    and specify the filter in the \uicontrol {Active filter}
    field. For more information about GTest filters, see
    \l{https://github.com/google/googletest/blob/master/docs/advanced.md#running-a-subset-of-the-tests}
    {Running a Subset of the Tests}.

    \section2 Specifying Settings for Running Boost Tests

    \list 1
        \li To specify settings for running Boost tests, select \uicontrol Edit
            > \uicontrol Preferences > \uicontrol {Testing} >
            \uicontrol {Boost Test}.
            \image qtcreator-preferences-testing-boosttest.webp {Boost Test tab in Testing preferences}
        \li In the \uicontrol {Log format} field, select the error report
            format to specify the type of events to record in the
            test report.
        \li In the \uicontrol {Report level} field, select the verbosity level
            of the test result report. Select \uicontrol No if you do not want
            a report.
        \li Select the \uicontrol Randomize check box to execute the tests in
            a random order, using the seed specified in the \uicontrol Seed
            field for initializing the randomizer. The value 0 means no
            randomization, the value 1 uses the current time, and any other
            value generates a random seed.
        \li Select the \uicontrol {Catch system errors} check box to catch
            system errors.
        \li Select the \uicontrol {Floating point exceptions} check box to
            detect floating point exceptions.
        \li Select the \uicontrol {Detect memory leaks} check box to detect
            memory leaks.
    \endlist

    \section2 Specifying Settings for Running Catch2 Tests
    \list 1
        \li To specify settings for running Catch2 tests, select
            \uicontrol Edit > \uicontrol Preferences > \uicontrol {Testing} >
            \uicontrol {Catch Test}.
            \image qtcreator-preferences-testing-catchtest.webp {Catch Test tab in Testing preferences}
        \li Select the \uicontrol {Show success} check box to show succeeding
            expressions as well. By default Catch2 will print only fails.
        \li Select the \uicontrol {Break on failure while debugging} check box
            to turn failures into debugger breakpoints.
        \li Select the \uicontrol {Skip throwing assertions} check box to skip
            any assertion that test for throwing an exception.
        \li Select the \uicontrol {Visualize whitespace} check box to turn
            whitespace into escape sequences.
        \li Select the \uicontrol {Warn on empty tests} check box to get a
            warning when a test case does not check any assertion.
        \li Select the \uicontrol {Abort after} check box to abort the test
            after the number of failures specified inside the spin box.
        \li Select the \uicontrol {Benchmark samples} check box to specify
            the number of samples to collect while running benchmarks.
        \li Select the \uicontrol {Benchmark resamples} check box to specify
            the number of resamples to use for statistical bootstrapping after
            benchmarking.
        \li Select the \uicontrol {Benchmark confidence interval} check box
            to specify the confidence interval for statistical bootstrapping.
        \li Select the \uicontrol {Benchmark warmup time} check box to specify
            the warmup time for each test before benchmarking starts.
        \li Select the \uicontrol {Disable analysis} check box to disable
            statistical analysis and bootstrapping.
    \endlist

    \section2 Specifying Settings for Running CTest-Based Tests
    \list 1
        \li To specify settings for running CTest-based tests, select
        \uicontrol Edit > \uicontrol Preferences > \uicontrol {Testing} >
        \uicontrol {CTest}.
            \image qtcreator-preferences-testing-ctest.webp {CTest tab in Testing preferences}
        \li Select the \uicontrol {Output on failure} check box to show test
            specific output if a test fails. Contrary to the CTest default
            this is enabled by default.
        \li Select \uicontrol {Schedule random} to execute the tests in
            random order.
        \li Select \uicontrol {Stop on failure} to automatically stop the
            test execution on the first failing test.
        \li In the \uicontrol {Output mode} field, select the verbosity level
            of the CTest output.
            \note This only influences the output on the text display.
        \li Select \uicontrol {Repeat tests} if you want to re-run tests
            under certain circumstances.
        \li In the \uicontrol {Repetition mode} field, select the mode for
            re-running tests. Set the maximum count for repeating a test in
            the \uicontrol {Count} field.
        \li Select \uicontrol {Run in parallel} to run the tests in parallel
            using the specified number of \uicontrol {Jobs}.
        \li Select \uicontrol {Test load} to limit the parallel
            execution. CTest will not start a new test if it would cause the
            CPU load to pass the threshold set in \uicontrol {Threshold}.
    \endlist

    \section1 Viewing Test Output

    The \l{Viewing Output}{Test Results} view shows Qt and Qt Quick test results
    in XML format and other test results in plain text format.

    \section2 Qt Test Output

    XML can be parsed more easily and reliably than plain text. However, if a Qt
    or Qt Quick test crashes, it might not produce complete XML
    code that can be parsed, which might lead to information loss. You might see
    the lost information when viewing the results as plain text.

    To view the
    results of Qt and Qt Quick tests as plain text, select \uicontrol Edit >
    \uicontrol Preferences > \uicontrol {Testing} > \uicontrol {Qt Test}, and
    then deselect the \uicontrol {Use XML output} check box. Then select the
    \inlineimage icons/text.png
    (\uicontrol {Switch Between Visual and Text Display}) button in
    \uicontrol {Test Results} to switch to the text display.

    \section2 Summary of Messages

    The following table lists the messages that \uicontrol {Test Results}
    displays:

    \table
        \header
            \li Result
            \li Description
        \row
            \li BENCH
            \li Benchmark test.
        \row
            \li DEBUG
            \li Debug message.
        \row
            \li FAIL
            \li Test case failed. Double-click the line for more information.
        \row
            \li FATAL
            \li A fatal error occurred that stops the test case from being run,
                for example.
        \row
            \li INFO
            \li Informative message.
        \row
            \li INTERNAL
            \li Internal message.
        \row
            \li PASS
            \li Test case passed.
        \row
            \li SKIP
            \li Test case was skipped.
        \row
            \li SYSTEM
            \li An error message received from or influenced by the OS.
        \row
            \li XFAIL
            \li Test case is expected to fail, so it is marked by using the
                QEXPECT_FAIL macro. If the test case passes instead, an
                unexpected pass (XPASS) is written to the test log.
        \row
            \li XPASS
            \li Test case passed even though it was expected to fail.
        \row
            \li WARN
            \li Warning message.

    \endtable

    To view only messages of a particular type, select
    \inlineimage icons/filtericon.png
    (\uicontrol {Filter Test Results}), and then select the types of messages to
    show. To show all messages, select \uicontrol {Check All Filters}. To
    deselect all message types, select \uicontrol {Uncheck All Filters}.

    Since Qt 5.4, you can add a BLACKLIST file for tests. It is mainly used
    internally by the Qt CI system.

    \table
        \header
            \li Result
            \li Description
        \row
            \li BFAIL
            \li Blacklisted test case failed.
        \row
            \li BPASS
            \li Blacklisted test case passed.
        \row
            \li BXFAIL
            \li Blacklisted test case failed but was marked to be expected to fail.
        \row
            \li BXPASS
            \li Blacklisted test case passed even though it was expected to fail.

    \endtable
*/