summaryrefslogtreecommitdiff
path: root/tests/README
diff options
context:
space:
mode:
authorDaniel Stenberg <daniel@haxx.se>2020-09-30 15:07:59 +0200
committerDaniel Stenberg <daniel@haxx.se>2020-09-30 22:45:45 +0200
commit422f610b409c731aaa5b16463e94dca55c7f0b97 (patch)
tree19b5cb7a40c1cccd4c6cb1caf550e76c12db116b /tests/README
parentd0c77730ed2056b05e5a379f7765d51cac4c23cc (diff)
downloadcurl-422f610b409c731aaa5b16463e94dca55c7f0b97.tar.gz
tests/README: convert to markdown
Closes #6028
Diffstat (limited to 'tests/README')
-rw-r--r--tests/README277
1 files changed, 0 insertions, 277 deletions
diff --git a/tests/README b/tests/README
deleted file mode 100644
index 209887dfc..000000000
--- a/tests/README
+++ /dev/null
@@ -1,277 +0,0 @@
- _ _ ____ _
- ___| | | | _ \| |
- / __| | | | |_) | |
- | (__| |_| | _ <| |___
- \___|\___/|_| \_\_____|
-
-The curl Test Suite
-
- 1. Running
- 1.1 Requires to run
- 1.2 Port numbers used by test servers
- 1.3 Test servers
- 1.4 Run
- 1.5 Shell startup scripts
- 1.6 Memory test
- 1.7 Debug
- 1.8 Logs
- 1.9 Test input files
- 1.10 Code coverage
- 1.11 Remote testing
-
- 2. Numbering
- 2.1 Test case numbering
-
- 3. Write tests
- 3.1 test data
- 3.2 curl tests
- 3.3 libcurl tests
- 3.4 unit tests
-
- 4. TODO
- 4.1 More protocols
- 4.2 SOCKS auth
-
-==============================================================================
-
-1. Running
-
- 1.1 Requires to run
-
- perl (and a unix-style shell)
- python (and a unix-style shell, for SMB and TELNET tests)
- python-impacket (for SMB tests)
- diff (when a test fails, a diff is shown)
- stunnel (for HTTPS and FTPS tests)
- OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
- nghttpx (for HTTP/2 tests)
- nroff (for --manual tests)
-
- 1.1.1 Installation of python-impacket
-
- The Python-based test servers support both recent Python 2 and 3.
- You can figure out your default Python interpreter with python -V
-
- Please install python-impacket in the correct Python environment.
- You can use pip or your OS' package manager to install 'impacket'.
-
- On Debian/Ubuntu the package names are:
- Python 2: 'python-impacket'
- Python 3: 'python3-impacket'
-
- On FreeBSD the package names are:
- Python 2: 'py27-impacket'
- Python 3: 'py37-impacket'
-
- On any system where pip is available:
- Python 2: 'pip2 install impacket'
- Python 3: 'pip3 install impacket'
-
- You may also need to manually install the Python package 'six'
- as that may be a missing requirement for impacket on Python 3.
-
- 1.2 Port numbers used by test servers
-
- Tests are written to use as few fixed fixed port numbers as possible and all
- tests should be written to use suitable variables instead of port numbers so
- that test cases continue to work independent on what port numbers the test
- servers actually use.
-
- The remaining fixed-port test servers that are still used, use the port
- range 8890 - 8904 by default, but can be moved with runtests' -b option.
-
- See the FILEFORMAT for a listing of existing port number variables.
-
- 1.3 Test servers
-
- The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
- servers on the ports listed above to which it makes requests. For SSL tests,
- it runs stunnel to handle encryption to the regular servers. For SSH, it
- runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
- the SOCKS functionality and requires a SSH client and server.
-
- The base port number (8990), which all the individual port numbers are
- indexed from, can be set explicitly using runtests.pl' -b option to allow
- running more than one instance of the test suite simultaneously on one
- machine, or just move the servers in case you have local services on any of
- those ports.
-
- The HTTP server supports listening on a Unix domain socket, the default
- location is 'http.sock'.
-
- 1.4 Run
-
- './configure && make && make test'. This builds the test suite support code
- and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
- variables of that script in case you have some specific needs, or run the
- script manually (after the support code has been built).
-
- The script breaks on the first test that doesn't do OK. Use -a to prevent
- the script from aborting on the first error. Run the script with -v for more
- verbose output. Use -d to run the test servers with debug output enabled as
- well. Specifying -k keeps all the log files generated by the test intact.
-
- Use -s for shorter output, or pass test numbers to run specific tests only
- (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
- ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
- 3 to 9. Any test numbers starting with ! are disabled, as are any test
- numbers found in the files data/DISABLED or data/DISABLED.local (one per
- line). The latter is meant for local temporary disables and will be ignored
- by git.
-
- When -s is not present, each successful test will display on one line the
- test number and description and on the next line a set of flags, the test
- result, current test sequence, total number of tests to be run and an
- estimated amount of time to complete the test run. The flags consist of
- these letters describing what is checked in this test:
-
- s stdout
- d data
- u upload
- p protocol
- o output
- e exit code
- m memory
- v valgrind
-
- 1.5 Shell startup scripts
-
- Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
- influenced by the output of system wide or user specific shell startup
- scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
- output text messages or escape sequences on user login. When these shell
- startup messages or escape sequences are output they might corrupt the
- expected stream of data which flows to the sftp-server or from the ssh
- client which can result in bad test behaviour or even prevent the test
- server from running.
-
- If the test suite ssh or sftp server fails to start up and logs the message
- 'Received message too long' then you are certainly suffering the unwanted
- output of a shell startup script. Locate, cleanup or adjust the shell
- script.
-
- 1.6 Memory test
-
- The test script will check that all allocated memory is freed properly IF
- curl has been built with the CURLDEBUG define set. The script will
- automatically detect if that is the case, and it will use the
- 'memanalyze.pl' script to analyze the memory debugging output.
-
- Also, if you run tests on a machine where valgrind is found, the script will
- use valgrind to run the test with (unless you use -n) to further verify
- correctness.
-
- runtests.pl's -t option will enable torture testing mode, which runs each
- test many times and makes each different memory allocation fail on each
- successive run. This tests the out of memory error handling code to ensure
- that memory leaks do not occur even in those situations. It can help to
- compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
- ensure that the memory log file is properly written even if curl crashes.
-
- 1.7 Debug
-
- If a test case fails, you can conveniently get the script to invoke the
- debugger (gdb) for you with the server running and the exact same command
- line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
- then just type 'run' in the debugger to perform the command through the
- debugger.
-
- 1.8 Logs
-
- All logs are generated in the log/ subdirectory (it is emptied first in the
- runtests.pl script). Use runtests.pl -k to force it to keep the temporary
- files after the test run since successful runs will clean it up otherwise.
-
- 1.9 Test input files
-
- All test cases are put in the data/ subdirectory. Each test is stored in the
- file named according to the test number.
-
- See FILEFORMAT for the description of the test case files.
-
- 1.10 Code coverage
-
- gcc provides a tool that can determine the code coverage figures for
- the test suite. To use it, configure curl with
- CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
- and torture tests to get more full coverage, i.e. do:
-
- make test
- make test-torture
-
- The graphical tool ggcov can be used to browse the source and create
- coverage reports on *NIX hosts:
-
- ggcov -r lib src
-
- The text mode tool gcov may also be used, but it doesn't handle object files
- in more than one directory very well.
-
- 1.11 Remote testing
-
- The runtests.pl script provides some hooks to allow curl to be tested on a
- machine where perl can not be run. The test framework in this case runs on
- a workstation where perl is available, while curl itself is run on a remote
- system using ssh or some other remote execution method. See the comments at
- the beginning of runtests.pl for details.
-
-2. Numbering
-
- 2.1 Test case numbering
-
- Test cases used to be numbered by category, but the ranges filled
- up. Subsets of tests can now be selected by passing keywords to the
- runtests.pl script via the make TFLAGS variable.
-
- New tests should now be added by finding a free number in
- tests/data/Makefile.inc.
-
-3. Write tests
-
- Here's a quick description on writing test cases. We basically have three
- kinds of tests: the ones that test the curl tool, the ones that build small
- applications and test libcurl directly and the unit tests that test
- individual (possibly internal) functions.
-
- 3.1 test data
-
- Each test has a master file that controls all the test data. What to read,
- what the protocol exchange should look like, what exit code to expect and
- what command line arguments to use etc.
-
- These files are tests/data/test[num] where [num] is described in section 2
- of this document, and the XML-like file format of them is described in the
- separate tests/FILEFORMAT document.
-
- 3.2 curl tests
-
- A test case that runs the curl tool and verifies that it gets the correct
- data, it sends the correct data, it uses the correct protocol primitives
- etc.
-
- 3.3 libcurl tests
-
- The libcurl tests are identical to the curl ones, except that they use a
- specific and dedicated custom-built program to run instead of "curl". This
- tool is built from source code placed in tests/libtest and if you want to
- make a new libcurl test that is where you add your code.
-
- 3.4 unit tests
-
- Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
- There's a tests/unit/README describing the specific set of checks and macros
- that may be used when writing tests that verify behaviors of specific
- individual functions.
-
- The unit tests depend on curl being built with debug enabled.
-
-4. TODO
-
- 4.1 More protocols
-
- Add tests for TELNET, LDAP, DICT...
-
- 4.2 SOCKS auth
-
- SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
- test mechanism) doesn't support them