From 39081b0c409284b8d9365597f3774bc20d3fe3cd Mon Sep 17 00:00:00 2001 From: Adrian Johnson Date: Sun, 1 Jan 2023 16:27:53 +1030 Subject: Add README.meson file for meson test --- test/README.meson | 329 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 329 insertions(+) create mode 100644 test/README.meson (limited to 'test') diff --git a/test/README.meson b/test/README.meson new file mode 100644 index 000000000..26b483973 --- /dev/null +++ b/test/README.meson @@ -0,0 +1,329 @@ +How to use cairo's test suite +============================= +Using this test should be as simple as running: + + meson test -C $builddir + +assuming that the cairo distribution in the directory above has been +configured and built in $builddir. The test suite here will use the +locally compiled library rather than installed version. + +The results of the test suite run are summarized in an index.html file +written to $builddir, which, when viewed in a web browser makes it +quite easy to visually see any failed renderings alongside the +corresponding reference image, (and a diff image as well). + +As some browsers do not permit Javascript to read from the local +filesystem, the view-test-results.py script in the $builddir can used +to view the results. It starts a http server serving the current +directory before displaying the test results in a browser. + +The test suite needs to be run before any code is committed and before +any release. See below for hints and rules governing the use of the suite. + +Running specific tests +---------------------- +During development it is desirable to only run a single test or groups +of related tests. The test suite is built as a single binary, +"cairo-test-suite", which allows you to choose individual or +categories of tests to run. + +The cairo-test-suite executable needs to be run in $builddir/test in +order to find the image conversion executables. Simply running the +executable will run all tests. eg + + cd $builddir/test + ./cairo-test-suite + +Individual tests can be run be specifying the tests names as arguments +to cairo-test-suite. For example, to run specific tests: + + ./cairo-test-suite record-neg-extents-unbounded record-neg-extents-bounded + +Or if you want to run all paint.* related tests you can use: + + ./cairo-test-suite paint + +Or if you want to check the current status of known failures: + + ./cairo-test-suite XFAIL + +Or to run a subset of tests, use the -k option to run only the tests +that include the given keyword: + + ./cairo-test-suite -k downscale + +The -l option lists all tests available. For convenience, the file +"completion.bash" in $builddir can be sourced to provide bash +completion of the test names in cairo-test-suite. + +By default, each test that is run will be tested on all backends that +have been built. The CAIRO_TEST_TARGET environment variable can be +used to select specific backends. For example: + + CAIRO_TEST_TARGET=pdf ./cairo-test-suite text-rotate + +Each test is run twice on each backend, once for each format. The +CAIRO_TEST_TARGET_FORMAT environment variable selects a single +format. The formats are "rgba" and "rgb". For example to run the +text-rotate test only on the image backend and only for the "rgba" +format: + + CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba ./cairo-test-suite text-rotate + +Running tests under modified environments or tools +-------------------------------------------------- +To run tests under in gdb, one can simply run the cairo-suite-suite in +gdb. Usually a specific test will be desired. eg + + CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba gdb --args ./cairo-test-suite text-rotate + +The cairo test suite runs each test in a child process so that crashes +do not stop the test. When running the test suite under tools, the "-f" +option disables forking. eg to run a test under valgrind: + + CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba valgrind ./cairo-test-suite -f text-rotate + +The cairo test suite detects if it is running under gdb and disables +forking. Hence the reason the gdb example above did not use "-f". + +Complete list of test options +----------------------------- +cairo-test-suite options: + + ./cairo-test-suite [-afkxsl] [test-names|keywords ...] + + -a all; run the full set of tests. By default the test suite + skips similar surface and device offset testing. + -f foreground; do not fork + -k match tests by keyword + -l list only; just list selected test case names without executing + -s include slow, long running tests + -x exit on first failure + +Environment Variables: + +CAIRO_TEST_TARGET + Run the test only the the listed targets. eg + CAIRO_TEST_TARGET="image" + CAIRO_TEST_TARGET="pdf,ps3,ps2" + +CAIRO_TEST_TARGET_FORMAT + Run the test only the the listed targets. eg + CAIRO_TEST_TARGET_FORMAT="rgba" + CAIRO_TEST_TARGET_FORMAT="rgba,rgb" + +CAIRO_TEST_MODE + May contain a list of one or more of the following options + "full" - equivalent to the "-a" option + "similar" + "offset" + "scale" + "foreground" - equivalent to the "-f" option + "exit-on-failure" - equivalent to the "-x" option + eg + CAIRO_TEST_MODE="full,foreground" + +CAIRO_TESTS + A list of test to run. This is equivalebt to listing the tests as + arguments to cairo=test-suite. eg + CAIRO_TESTS="record-neg-extents-unbounded record-neg-extents-bounded" + The tests may be separated by anyh of " \t,:;". + +srcdir + The test suite needs to find the "test" directory in the source + tree. The srcdir environment variable can be used to specify the + location of this directory. If this environment variable is not set, + the binary looks for the directory "srcdir" in the current + directory. The meson build system symlinks "srcdir" in the + $builddir/test directory to the "test" directory in the source + tree. If this is not found the binary defaults to the current + directory. + +CAIRO_REF_DIR + The directory containing the reference images. By default the test + suite looks for "reference" in $srcdir. For convenience, you can set + CAIRO_REF_DIR to point at a previous test directory, relative to the + current test directory, and any previous output will be used by + preference as reference images. + +CAIRO_TEST_TIMEOUT + The maximum time, in seconds, an indivdual test may run before it is + stopped. The default is 60 seconds. + +Getting the elusive zero failures +--------------------------------- +It's generally been very difficult to achieve a test run with zero +failures. The difficulties stem from the various versions of the many +libraries that the test suite depends on, (it depends on a lot more +than cairo itself), as well as fonts and other system-specific +settings. If your system differs significantly from the system on +which the reference images were generated, then you will likely see +the test suite reporting "failures", (even if cairo is working just +fine). + +Since the test suite must pass the Continuous Integration test in the +Gitlab instance hosting the cairo repository, the easiest way to get a +new or updated test to pass is to extract the test output from the +cairo gitlab CI to use as reference images. + +What if I can't make my system match? +------------------------------------- +For one reason or another, you may be unable to get a clean run of the +test suite even if cairo is working properly, (for example, you might +be on a system without freetype). In this case, it's still useful to +be able to determine if code changes you make to cairo result in any +regressions to the test suite. But it's hard to notice regressions if +there are many failures both before and after your changes. + +For this scenario, you can capture the output of a run of the test +suite before your changes, and then use the CAIRO_REF_DIR environment +variable to use that output as the reference images for a run after +your changes. The process looks like this: + + # Before code change there may be failures we don't care about + cd $builddir/test + ./cairo-test-suite + + # Let's save those output images + mkdir /some/directory/ + cp -r output /some/directory/ + + # hack, hack, hack + + # Now to see if nothing changed: + CAIRO_REF_DIR=/some/directory/ ./cairo-test-suite + +Best practices for cairo developers +=================================== +If we all follow the guidelines below, then both the test suite and +cairo itself will stay much healthier, and we'll all have a lot more +fun hacking on cairo. + +Before creating a Merge Request +------------------------------- +After pushing your branch to your cairo fork on gitlab.freedesktop.org, +check that the CI passes. A MR cannot be merged until the CI passes. + +It is likely that you will need to download test output images from +the Gitlab CI to use as reference images in order to get CI to pass. + +The reference image filenames have the form: + +.*.ref.png for correct images that pass. + +.*.xfail.png for incorrect images due to bugs. + +The .ref.png is the most general image that all backends +and formats for the test will be required to match. + +When comparing test output to the reference images, the test suite +performs a perceptual diff which permits some minor pixel differences +that are not visually discernible. Some targets may have differences +that exceed the threshold of the perceptual diff but still appear the +same as the reference image when eyeballed. In these cases, target +specific reference images can be created that override the general +reference image for the specific target. + +Reference images for specific backends can be created using filenames of the form: + ..ref.png + +eg clip.pdf.ref.png + +The target name "ps" can be used for both "ps2" and "ps3" targets. + +Reference images for specific formats can be created using filenames of the form: + ...ref.png + +eg + clip.pdf.argb32.ref.png + clip.pdf.rgb24.ref.png + +The recommended practice for creating reference images for a new test is: + +1. Create a .ref.png reference image from the *.image.argb32.out.png output. + +2. Push to gitlab and check the CI result. + +3. If the image target fails, copy the *.image.argb32.out.png output +from CI and use as the reference image. + +4. Once the image target passes in CI, check that the other targets +pass. If any fail, create *..ref.png reference images if the +output appears correct. + +5. If a target output is incorrect, try to determine if the cairo +target output is correct. eg view pdf files in Adobe Reader. If the +cairo target output is correct, it can be assumed the image converson +from target to png is buggy. File a bug against the conversion tool +and create a *..xfail.png reference image. + +In all cases, before creating a reference image from test output, the +image should be viewed to ensure that it is correct. + +When a new bug is found +----------------------- +A new test case should be added by imitating the style of an existing +test. This means adding the following files: + + new-bug.c + reference/new-bug.ref.png + reference/new-bug.xfail.png + +Where new-bug.c is a minimal program to demonstrate the bug, following +the style of existing tests. The new-bug.ref.png image should contain +the desired result of new-bug.c if the bug were fixed while +new-bug.xfail.png contains the current results of the test. + +"meson.build" should be edited by adding new-bug.c to test_sources. +And last but not least, don't forget to "git add" the new files. + +When a new feature is added +--------------------------- +It's important for the regression suite to keep pace with development +of the library. So a new test should be added for each new feature. +The work involved is similar the work described above for new bugs. +The only distinction is that the test is expected to pass so it +should not need a new-bug.xfail.png file. + +While working on a test +----------------------- +Before a bugfix or feature is ready, it may be useful to compare +output from different builds. For convenience, you can set +CAIRO_REF_DIR to point at a previous test directory, relative +to the current test directory, and any previous output will be +used by preference as reference images. + +When a bug is fixed +------------------- +The fix should be verified by running the test suite which should +result in an "unexpected pass" for the test of interest. Rejoice as +appropriate, then remove the relevant xfail.png file from git. + +Before releasing +---------------- +All tests should return a result of PASS for all supported (those enabled by +default) backends, meaning all known bugs are fixed, resulting in the happy +message: + + All XX tests passed + +Some notes on limitations in poppler +==================================== +One of the difficulties of our current test infrastructure is that we +rely on external tools to convert cairo's vector output (PDF, +PostScript, and SVG), into an image that can be used for the image +comparison. This means that any bugs in that conversion tool will +result in false negatives in the test suite. + +Here's the rule: If a cairo-pdf test reports a failure, but viewing +the resulting PDF file with acroread suggests that the PDF itself is +correct, then there's likely a bug in poppler. In this case, we can +simply report the poppler bug. Once we've done this, we can capture +poppler's buggy output as a pdf-specific reference image (as +reference/*.xfail.png) so that the test suite will regard the test as +passing, (and we'll ensure there is no regression). + +Once the poppler bug gets fixed, the test suite will start reporting a +false negative again, and this will be easy to fix by simply removing +the pdf-specific reference image. -- cgit v1.2.1