diff options
Diffstat (limited to 'tests/README')
| -rw-r--r-- | tests/README | 277 |
1 files changed, 0 insertions, 277 deletions
diff --git a/tests/README b/tests/README deleted file mode 100644 index 209887dfc..000000000 --- a/tests/README +++ /dev/null @@ -1,277 +0,0 @@ - _ _ ____ _ - ___| | | | _ \| | - / __| | | | |_) | | - | (__| |_| | _ <| |___ - \___|\___/|_| \_\_____| - -The curl Test Suite - - 1. Running - 1.1 Requires to run - 1.2 Port numbers used by test servers - 1.3 Test servers - 1.4 Run - 1.5 Shell startup scripts - 1.6 Memory test - 1.7 Debug - 1.8 Logs - 1.9 Test input files - 1.10 Code coverage - 1.11 Remote testing - - 2. Numbering - 2.1 Test case numbering - - 3. Write tests - 3.1 test data - 3.2 curl tests - 3.3 libcurl tests - 3.4 unit tests - - 4. TODO - 4.1 More protocols - 4.2 SOCKS auth - -============================================================================== - -1. Running - - 1.1 Requires to run - - perl (and a unix-style shell) - python (and a unix-style shell, for SMB and TELNET tests) - python-impacket (for SMB tests) - diff (when a test fails, a diff is shown) - stunnel (for HTTPS and FTPS tests) - OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests) - nghttpx (for HTTP/2 tests) - nroff (for --manual tests) - - 1.1.1 Installation of python-impacket - - The Python-based test servers support both recent Python 2 and 3. - You can figure out your default Python interpreter with python -V - - Please install python-impacket in the correct Python environment. - You can use pip or your OS' package manager to install 'impacket'. - - On Debian/Ubuntu the package names are: - Python 2: 'python-impacket' - Python 3: 'python3-impacket' - - On FreeBSD the package names are: - Python 2: 'py27-impacket' - Python 3: 'py37-impacket' - - On any system where pip is available: - Python 2: 'pip2 install impacket' - Python 3: 'pip3 install impacket' - - You may also need to manually install the Python package 'six' - as that may be a missing requirement for impacket on Python 3. - - 1.2 Port numbers used by test servers - - Tests are written to use as few fixed fixed port numbers as possible and all - tests should be written to use suitable variables instead of port numbers so - that test cases continue to work independent on what port numbers the test - servers actually use. - - The remaining fixed-port test servers that are still used, use the port - range 8890 - 8904 by default, but can be moved with runtests' -b option. - - See the FILEFORMAT for a listing of existing port number variables. - - 1.3 Test servers - - The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone - servers on the ports listed above to which it makes requests. For SSL tests, - it runs stunnel to handle encryption to the regular servers. For SSH, it - runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform - the SOCKS functionality and requires a SSH client and server. - - The base port number (8990), which all the individual port numbers are - indexed from, can be set explicitly using runtests.pl' -b option to allow - running more than one instance of the test suite simultaneously on one - machine, or just move the servers in case you have local services on any of - those ports. - - The HTTP server supports listening on a Unix domain socket, the default - location is 'http.sock'. - - 1.4 Run - - './configure && make && make test'. This builds the test suite support code - and invokes the 'runtests.pl' perl script to run all the tests. Edit the top - variables of that script in case you have some specific needs, or run the - script manually (after the support code has been built). - - The script breaks on the first test that doesn't do OK. Use -a to prevent - the script from aborting on the first error. Run the script with -v for more - verbose output. Use -d to run the test servers with debug output enabled as - well. Specifying -k keeps all the log files generated by the test intact. - - Use -s for shorter output, or pass test numbers to run specific tests only - (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case - ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from - 3 to 9. Any test numbers starting with ! are disabled, as are any test - numbers found in the files data/DISABLED or data/DISABLED.local (one per - line). The latter is meant for local temporary disables and will be ignored - by git. - - When -s is not present, each successful test will display on one line the - test number and description and on the next line a set of flags, the test - result, current test sequence, total number of tests to be run and an - estimated amount of time to complete the test run. The flags consist of - these letters describing what is checked in this test: - - s stdout - d data - u upload - p protocol - o output - e exit code - m memory - v valgrind - - 1.5 Shell startup scripts - - Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly - influenced by the output of system wide or user specific shell startup - scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which - output text messages or escape sequences on user login. When these shell - startup messages or escape sequences are output they might corrupt the - expected stream of data which flows to the sftp-server or from the ssh - client which can result in bad test behaviour or even prevent the test - server from running. - - If the test suite ssh or sftp server fails to start up and logs the message - 'Received message too long' then you are certainly suffering the unwanted - output of a shell startup script. Locate, cleanup or adjust the shell - script. - - 1.6 Memory test - - The test script will check that all allocated memory is freed properly IF - curl has been built with the CURLDEBUG define set. The script will - automatically detect if that is the case, and it will use the - 'memanalyze.pl' script to analyze the memory debugging output. - - Also, if you run tests on a machine where valgrind is found, the script will - use valgrind to run the test with (unless you use -n) to further verify - correctness. - - runtests.pl's -t option will enable torture testing mode, which runs each - test many times and makes each different memory allocation fail on each - successive run. This tests the out of memory error handling code to ensure - that memory leaks do not occur even in those situations. It can help to - compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to - ensure that the memory log file is properly written even if curl crashes. - - 1.7 Debug - - If a test case fails, you can conveniently get the script to invoke the - debugger (gdb) for you with the server running and the exact same command - line parameters that failed. Just invoke 'runtests.pl <test number> -g' and - then just type 'run' in the debugger to perform the command through the - debugger. - - 1.8 Logs - - All logs are generated in the log/ subdirectory (it is emptied first in the - runtests.pl script). Use runtests.pl -k to force it to keep the temporary - files after the test run since successful runs will clean it up otherwise. - - 1.9 Test input files - - All test cases are put in the data/ subdirectory. Each test is stored in the - file named according to the test number. - - See FILEFORMAT for the description of the test case files. - - 1.10 Code coverage - - gcc provides a tool that can determine the code coverage figures for - the test suite. To use it, configure curl with - CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal - and torture tests to get more full coverage, i.e. do: - - make test - make test-torture - - The graphical tool ggcov can be used to browse the source and create - coverage reports on *NIX hosts: - - ggcov -r lib src - - The text mode tool gcov may also be used, but it doesn't handle object files - in more than one directory very well. - - 1.11 Remote testing - - The runtests.pl script provides some hooks to allow curl to be tested on a - machine where perl can not be run. The test framework in this case runs on - a workstation where perl is available, while curl itself is run on a remote - system using ssh or some other remote execution method. See the comments at - the beginning of runtests.pl for details. - -2. Numbering - - 2.1 Test case numbering - - Test cases used to be numbered by category, but the ranges filled - up. Subsets of tests can now be selected by passing keywords to the - runtests.pl script via the make TFLAGS variable. - - New tests should now be added by finding a free number in - tests/data/Makefile.inc. - -3. Write tests - - Here's a quick description on writing test cases. We basically have three - kinds of tests: the ones that test the curl tool, the ones that build small - applications and test libcurl directly and the unit tests that test - individual (possibly internal) functions. - - 3.1 test data - - Each test has a master file that controls all the test data. What to read, - what the protocol exchange should look like, what exit code to expect and - what command line arguments to use etc. - - These files are tests/data/test[num] where [num] is described in section 2 - of this document, and the XML-like file format of them is described in the - separate tests/FILEFORMAT document. - - 3.2 curl tests - - A test case that runs the curl tool and verifies that it gets the correct - data, it sends the correct data, it uses the correct protocol primitives - etc. - - 3.3 libcurl tests - - The libcurl tests are identical to the curl ones, except that they use a - specific and dedicated custom-built program to run instead of "curl". This - tool is built from source code placed in tests/libtest and if you want to - make a new libcurl test that is where you add your code. - - 3.4 unit tests - - Unit tests are tests in the 13xx sequence and they are placed in tests/unit. - There's a tests/unit/README describing the specific set of checks and macros - that may be used when writing tests that verify behaviors of specific - individual functions. - - The unit tests depend on curl being built with debug enabled. - -4. TODO - - 4.1 More protocols - - Add tests for TELNET, LDAP, DICT... - - 4.2 SOCKS auth - - SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the - test mechanism) doesn't support them |