summaryrefslogtreecommitdiff
path: root/tests/system/README
blob: 6a62bb8d056e055ca90be20f031d08f149bed2b3 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
Prerequisites - general information
-----------------------------------
Squish tests inside this folder have several prerequisites to get them running.

First - and most important - you have to own a valid Squish license. Currently it's recommended to use Squish 6.0.

Second - some of the test suites/test cases expect a build of Qt 4.8.7 to be available:
 1. Download the source code from:
    * Windows: http://download.qt.io/official_releases/qt/4.8/4.8.7/qt-everywhere-opensource-src-4.8.7.zip
    * Other: http://download.qt.io/official_releases/qt/4.8/4.8.7/qt-everywhere-opensource-src-4.8.7.tar.gz
 2. Extract the contents of the archive's directory qt-everywhere-opensource-src-4.8.7 to:
    * Windows: C:\Qt\Qt4.8.7
    * Other: $HOME/Qt4.8.7
 3. Apply the changes from patch.txt next to this README.
 4. In the directory you extracted the sources to, configure Qt:
    * Windows (MSVC2013 32 bit):
      .\configure.exe -opensource -developer-build -confirm-license -debug-and-release -nomake tests -nomake examples -nomake demos -no-webkit -no-phonon
    * Linux (gcc < 6):
      ./configure -opensource -developer-build -confirm-license -debug-and-release -nomake tests -nomake examples -nomake demos -no-webkit -no-phonon
    * macOS:
      ./configure -opensource -developer-build -confirm-license -debug-and-release -nomake tests -nomake examples -nomake demos -no-webkit -no-phonon -sdk <PATH_TO_INSTALLED_MACOSX_SDK>
 5. Make:
    * Windows (do not use jom):
      nmake
    * Other:
      make -j<number of available cores>

Third - some of the test suites/test cases expect Qt 5.3.1 (default toolchain), Qt 5.4.1 (gcc, Linux and Windows only) and Qt 5.6.1-1 (default toolchain)
installed in their default locations. On Linux/Mac this is ~/Qt5.x.1 and on Windows this is C:\Qt\Qt5.x.1. The default toolchains are gcc on Linux,
clang on Mac and MSVC2010 (Qt <= 5.5) or MSVC2013 (Qt > 5.5) on Windows. It's easiest to use default installations of the official opensource Qt packages.

On macOS it might be necessary to tweak some files of the Qt installation - depending on the installed Xcode and its installed SDKs.
If Xcode 8+ is installed you will need to update the file
~/Qt5.3.1/5.3/clang_64/mkspecs/features/mac/default_pre.prf
Change the line

 isEmpty($$list($$system("/usr/bin/xcrun -find xcrun 2>/dev/null")))

to

 isEmpty($$list($$system("/usr/bin/xcrun -find xcodebuild 2>/dev/null")))

Furthermore - depending on the installed SDK it might be necessary to update the file
~/Qt5.3.1/5.3/clang_64/mkspecs/qdevice.pri
Search for the following line

 !host_build:QMAKE_MAC_SDK = macosx10.8

Newer Xcode might miss the 10.8 SDK. Check
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs
which SDKs are installed and update the version number accordingly.

Fourth - you'll have to provide some additional repositories (and for the hooking into subprocesses even some more Squish bundles, see below).
These additional repositories are located inside ~/squish-data or C:\Users\<user>\squish-data (depending on the OS you're on).
You can also just provide them inside a different folder and specify the folder with the environment variable SYSTEST_SRCPATH.
This folder must contain the following:
 * a QtCreator repository (or source copy) of tag v4.2.2 named 'creator' including the submodule src/shared/qbs
 * a subfolder called 'creator-test-data'
 * a speedcrunch 0.11 repository (or source copy) inside 'creator-test-data' named 'speedcrunch'
 * additional Squish versions for hooking into subprocesses inside different folders inside 'creator-test-data' following the information below

Fifth - you'll have to make sure that some needed tools are available (no matter on which OS you're on).
 * cmake
 * wget or curl, capable of HTTPS
Normally it should be okay to just install them as usual and add their executables' path(s) to the PATH variable.

Sixth - Qt Creator should be built with ClangCodeModel plugin. How to do so, see QTCREATOR_REPO/src/plugins/clangcodemodel/README. Without the
plugin, the tests for ClangCodeModel will be skipped but will not cause failures.

On macOS make sure you are using the correct keyboard layout to avoid problems when using keyboard interaction. Tested and known to be
working would be 'U.S. International - PC', while pure 'U.S.' had problems.
Use the settings in System Preferences > Keyboard > Input Sources to install missing keyboard layouts and ensure you have set it as default.

Attention! If any of these prerequisites cannot be satisfied the tests will likely fail (or not run at all).


Prerequisites - hooking into subprocesses
-----------------------------------------
The usage of the hook-into subprocess has following prerequisites:
Either:
    * have no firewall at all enabled (sure that's a bad idea)
Or:
    * have the Windows Firewall enabled (no other firewalls are handled by the scripts)
    * run the Squish tests with administrator privileges
    * additionally the UAC should be disabled, too

Otherwise you'll have some trouble with popping up dialogs from the firewall.
If you're using a different firewall - try to figure out and add a rule for this.


Using the internal test data fallback
-------------------------------------

    * inside the shared_data folder a "standard" mapping file is located (qt_squish_mapping.tsv)
    * this file defines currently some Squish versions that are available for usage
    * all of the Squish versions mentioned in there should exist inside QtSDK/src/creator-test-data (see the file for naming the Squish versions) - of course this depends on the OS you're testing on

Attention! This file will be updated with more columns (and rows) for making it possible to also test Debug builds or 64bit builds. So, even the path entries currently listed could change when this feature is implemented.


Using QT_SQUISH_MAPFILE variable
--------------------------------

    * create a simple text file (UTF-8 encoded) that follows this scheme:

     SquishVersion QtVersion mkspec Path

    * SquishVersion: only major and minor number will be used (e.g. 4.1, 4.2)
    * QtVersion: only major and minor number will be used (e.g. 4.7, 4.8, 5.0)
    * mkspec: a string holding the mkspec as it appears inside QTSDK/mkspec (or inside Qt Creator)
    * Path: the path to the Squish directory that can be used with this combination of QtVersion and mkspec
    * between QtVersion and mkspec as well as between mkspec and Path use whitespaces only
    * lines STARTING with # within the first column will be ignored
    * path can contain spaces as well as ~ (for the home directory)
    * you can put any entry of a line into quotes (no matter whether single or double quotes)
    * a line holding a mapping MUST start with a QtVersion entry in the first column

Example 1: (using single space, no quoting)

#squishversion qtversion mkspec path
4.1<SP>4.7<SP>win32-g++<SP>C:\Tools\Squish_MinGW
4.1<SP>4.7<SP>win32-msvc2008<SP>C:\Tools\Squish_MSVC9
4.1<SP>4.7<SP>win32-msvc2010<SP>C:\Tools\Squish_MSVC10
4.2<SP>4.8<SP>win32-g++<SP>C:\Tools\Squish_MinGW
4.2<SP>4.8<SP>win32-msvc2008<SP>C:\Tools\Squish_MSVC9
4.2<SP>4.8<SP>win32-msvc2010<SP>C:\Tools\Squish_MSVC10

Example 2: (using mixed whitespaces, some quoting)

#squishversion<SP>qtversion<TAB>mkspec<TAB><TAB><SP>path
4.1<SP>"4.7"<SP><TAB>win32-g++<TAB><TAB>'C:\Tools\Squish_MinGW'
4.1<SP>'4.7'<SP><TAB>'win32-msvc2008'<TAB>"C:\Tools\Squish_MSVC9"
4.1<SP>4.7<SP><SP><TAB>win32-msvc2010<TAB>C:\Tools\Squish_MSVC10
4.2<SP>"4.8"<SP><TAB>"win32-g++"<SP><SP><TAB>"C:\Tools\Squish_MinGW"
4.2<SP>'4.8'<SP><TAB>win32-msvc2008<SP><TAB>'C:\Tools\Squish_MSVC9'
4.2<SP>'4.8'<SP><TAB>win32-msvc2010<SP><TAB>C:\Tools\Squish_MSVC10

Explanation: <SP> = space, <TAB> = tabulator

    * after creating this file put it somewhere and define environment variable QT_SQUISH_MAPFILE to point directly to the file
    * definition of this variable should be done BEFORE the test is started and can be done in various ways:
        * declare this variable as a system environment variable (how to do this depends on the OS you're using) [recommended way]
        * add this variable to the envvar file inside the directory holding the Squish test suites [please don't do this when pushing envvar back to git]
        * you can also modify the os.environ dict inside the test script(s) but this has to be done, before the test tries to read the variable [best place would be right before/after the call to startApplication()]

Hint: You can also use the provided tsv file as a template. Simply remove lines you're not needing and adjust the path entries to point to the correct paths on your system and you should be fine.

Attention! The format of the file might slightly change as soon also Debug builds will be easily testable (and maybe again for 64bit builds)


Preparation of the SQUISH directories
-------------------------------------
To make the hook-into sub-process really work you have to provide a modified Squish.

Steps to set up the Squish directories on Windows:
    * get the Squish version you need from froglogic.com
    * extract the archive to place of your choice
    * go to the bin directory of Squish and remove the following dll's (Windows-only):
        * Qt3Support.dll
        * QtCore4.dll
        * QtGui4.dll
        * QtNetwork4.dll
        * QtSql4.dll
        * QtXml4.dll
    * remove all folders from the Squish root folder except bin, etc, lib and python - and on
      Windows you have to keep the VC runtime folder (Microsoft.VC??.CRT)
      (if you're using other script languages you have to keep them as well)
    * you can remove regular files from Squish root as well
      (but it's recommended to keep the buildinfo.txt for later verifications)

You can additionally remove the complete ide folder as well as squishclassicide.exe and squishide.exe (to safe disk space).