summaryrefslogtreecommitdiff
path: root/chromium/docs/testing
diff options
context:
space:
mode:
authorAllan Sandfeld Jensen <allan.jensen@qt.io>2021-05-20 09:47:09 +0200
committerAllan Sandfeld Jensen <allan.jensen@qt.io>2021-06-07 11:15:42 +0000
commit189d4fd8fad9e3c776873be51938cd31a42b6177 (patch)
tree6497caeff5e383937996768766ab3bb2081a40b2 /chromium/docs/testing
parent8bc75099d364490b22f43a7ce366b366c08f4164 (diff)
downloadqtwebengine-chromium-189d4fd8fad9e3c776873be51938cd31a42b6177.tar.gz
BASELINE: Update Chromium to 90.0.4430.221
Change-Id: Iff4d9d18d2fcf1a576f3b1f453010f744a232920 Reviewed-by: Allan Sandfeld Jensen <allan.jensen@qt.io>
Diffstat (limited to 'chromium/docs/testing')
-rw-r--r--chromium/docs/testing/batching_instrumentation_tests.md4
-rw-r--r--chromium/docs/testing/chromeos_debugging_tips.md38
-rw-r--r--chromium/docs/testing/code_coverage.md11
-rw-r--r--chromium/docs/testing/code_coverage_in_gerrit.md2
-rw-r--r--chromium/docs/testing/gtest_flake_tips.md14
-rw-r--r--chromium/docs/testing/regression-test-selection.md11
-rw-r--r--chromium/docs/testing/testing_in_chromium.md2
-rw-r--r--chromium/docs/testing/web_platform_tests.md36
-rw-r--r--chromium/docs/testing/web_platform_tests_wptrunner.md8
-rw-r--r--chromium/docs/testing/web_test_expectations.md12
-rw-r--r--chromium/docs/testing/web_tests.md42
-rw-r--r--chromium/docs/testing/web_tests_in_content_shell.md4
12 files changed, 127 insertions, 57 deletions
diff --git a/chromium/docs/testing/batching_instrumentation_tests.md b/chromium/docs/testing/batching_instrumentation_tests.md
index ec96a3934a4..1c9ea9e3420 100644
--- a/chromium/docs/testing/batching_instrumentation_tests.md
+++ b/chromium/docs/testing/batching_instrumentation_tests.md
@@ -61,6 +61,10 @@ will run the suite in its own batch. This will reduce the complexity of managing
and leaking state from these tests as you only have to think about tests within
the suite. For smaller and less complex test suites, see Custom below.
+If you use different @Features annotations on test methods, you can use the
+@Batch.SplitByFeature annotation to run tests with different features in
+separate batches.
+
### Custom
This batching type is best for smaller and less complex test suites, that
diff --git a/chromium/docs/testing/chromeos_debugging_tips.md b/chromium/docs/testing/chromeos_debugging_tips.md
index afe581c4554..9cc3892acc8 100644
--- a/chromium/docs/testing/chromeos_debugging_tips.md
+++ b/chromium/docs/testing/chromeos_debugging_tips.md
@@ -35,6 +35,8 @@ system log filename. For instance, the previous `ui.WindowControl` failure
matches the [system_logs/chrome/chrome_20201029-195153] browser log, which
contains the culprit Chrome crash and backtrace.
+- **Symbolizing a browser crash dump**: See [below](#symbolizing-a-crash-dump).
+
### Disabling a test
There a couple ways to disable a test on Chrome's builders:
@@ -50,6 +52,40 @@ disabled tests for the step's GN target. For example, to disable a test in the
In both cases, please make sure a bug is filed for the test, and route it to
the appropriate owners.
+### Symbolizing a crash dump
+
+If a test fails due to a browser crash, there should be a Minidump crash report
+present in the test's isolated out under the prefix `crashes/chrome...`. These
+reports aren't very useful by themselves, but with a few commands you can
+symbolize the report locally to get insight into what conditions caused Chrome
+to crash.
+
+To do so, first download both the task's input isolate (this provides the
+symbols and the symbolizing tools) as well as the task's output isolate (this
+provides the crash reports). See the commands listed under the *Reproducing the
+task locally* section on the task page. For example, to download them for
+[this task](https://chrome-swarming.appspot.com/task?id=506a01dd12c8a610), `cd`
+into a tmp directory and run:
+```
+$CHROME_DIR/tools/luci-go/isolated download -I https://chrome-isolated.appspot.com --namespace default-gzip -isolated 64919fee8b02d826df2401544a9dc0f7dfa2172d -output-dir input
+python $CHROME_DIR/tools/swarming_client/swarming.py collect -S chrome-swarming.appspot.com 506a01dd12c8a610 --task-output-dir output
+```
+
+Once both isolates have been fetched you must then generate the breakpad
+symbols by pointing the `generate_breakpad_symbols.py` script to the input's
+build dir:
+```
+python input/components/crash/content/tools/generate_breakpad_symbols.py --symbols-dir symbols --build-dir input/out/Release/ --binary input/out/Release/chrome
+```
+
+That will generate the symbols in the `symbols/` dir. Then to symbolize a Chrome
+crash report present in the task's output (such as
+`chrome.20201211.041043.31022.5747.dmp`):
+```
+./input/out/Release/minidump_stackwalk output/0/crashes/chrome.20201211.041043.31022.5747.dmp symbols/
+```
+
+
### Running a test locally
To run a Tast test the same way it's ran on Chrome's builders:
@@ -87,7 +123,7 @@ To run a Tast test the same way it's ran on Chrome's builders:
[tests/ui.WindowControl/messages]: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=baefbcfd24c02b3ada4617d259dc6b4220b413b9&as=messages
[system_logs/chrome/chrome_20201029-195153]: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=272166c85f190c336a9885f0267cbdea912e31da&as=chrome_20201029-195153
[Tast attributes]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/test_attributes.md
-[this list]: https://codesearch.chromium.org/chromium/src/chromeos/BUILD.gn?rcl=7b0393a9091fd02edc9ae773739124f7be5a0782&l=242
+[this list]: https://codesearch.chromium.org/chromium/src/chromeos/tast_control.gni
[Simple Chrome]: https://chromium.googlesource.com/chromiumos/docs/+/master/simple_chrome_workflow.md
[deploy_chrome.py]: https://chromium.googlesource.com/chromiumos/docs/+/master/simple_chrome_workflow.md#Deploying-Chrome-to-the-device
[here]: https://chromium.googlesource.com/chromiumos/docs/+/master/cros_vm.md#in-simple-chrome
diff --git a/chromium/docs/testing/code_coverage.md b/chromium/docs/testing/code_coverage.md
index b3c9fff93c9..a97ed214575 100644
--- a/chromium/docs/testing/code_coverage.md
+++ b/chromium/docs/testing/code_coverage.md
@@ -124,7 +124,8 @@ by CQ bot. Or see this
The [coverage script] automates the process described below and provides a
one-stop service to generate code coverage reports locally in just one command.
-This script is currently supported on Linux, Mac, iOS and ChromeOS platforms.
+This script is currently supported on Android, Linux, Mac, iOS and ChromeOS
+platforms.
Here is an example usage:
@@ -185,10 +186,11 @@ hundred, resulting in the generation of a few hundred gigabytes’ raw
profiles. To limit the number of raw profiles, `%Nm` pattern in
`LLVM_PROFILE_FILE` environment variable is used to run tests in multi-process
mode, where `N` is the number of raw profiles. With `N = 4`, the total size of
-the raw profiles are limited to a few gigabytes.
+the raw profiles are limited to a few gigabytes. (If working on Android, the
+.profraw files will be located in ./out/coverage/coverage by default.)
```
-$ export LLVM_PROFILE_FILE=”out/report/crypto_unittests.%4m.profraw”
+$ export LLVM_PROFILE_FILE="out/report/crypto_unittests.%4m.profraw"
$ ./out/coverage/crypto_unittests
$ ls out/report/
crypto_unittests.3657994905831792357_0.profraw
@@ -233,6 +235,9 @@ $ llvm-cov show -output-dir=out/report -format=html \
out/coverage/crypto_unittests
```
+If creating a report for Android, the -object arg would be the lib.unstripped
+file, ie out/coverage/lib.unstripped/libcrypto_unittests__library.so
+
For more information on how to use llvm-cov, please refer to the [guide].
## Contacts
diff --git a/chromium/docs/testing/code_coverage_in_gerrit.md b/chromium/docs/testing/code_coverage_in_gerrit.md
index 44767b64e91..97d77adab14 100644
--- a/chromium/docs/testing/code_coverage_in_gerrit.md
+++ b/chromium/docs/testing/code_coverage_in_gerrit.md
@@ -67,7 +67,7 @@ in Gerrit.
[file a bug]: https://bugs.chromium.org/p/chromium/issues/entry?components=Infra%3ETest%3ECodeCoverage
[code-coverage group]: https://groups.google.com/a/chromium.org/forum/#!forum/code-coverage
[code_coverage.md]: code_coverage.md
-[clang_code_coverage_wrapper]: clang_code_coverage_wrapper.md
+[clang_code_coverage_wrapper]: https://chromium.googlesource.com/chromium/src/+/master/docs/clang_code_coverage_wrapper.md
[chromium-coverage Gerrit plugin]: https://chromium.googlesource.com/infra/gerrit-plugins/code-coverage/
[Chromium on Chromium OS]: https://chromium.googlesource.com/chromium/src/+/master/docs/chromeos_build_instructions.md
[Chromium on Linux]: https://chromium.googlesource.com/chromium/src/+/master/docs/linux/build_instructions.md
diff --git a/chromium/docs/testing/gtest_flake_tips.md b/chromium/docs/testing/gtest_flake_tips.md
index f9cf55bb40e..5077c81d74e 100644
--- a/chromium/docs/testing/gtest_flake_tips.md
+++ b/chromium/docs/testing/gtest_flake_tips.md
@@ -58,12 +58,20 @@ reproduce the flake with more information.
If the test is flakily timing out, consider any asynchronous code that may cause
race conditions, where the test subject may early exit and miss a callback, or
return faster than the test can start waiting for it (i.e. make sure event
-listeners are spawned before invoking the event).
+listeners are spawned before invoking the event). Make sure event listeners are
+for the proper event instead of a proxy (e.g. [Wait for the correct event in
+test](https://chromium.googlesource.com/chromium/src/+/6da09f7510e94d2aebbbed13b038d71c511d6cbc)).
+
+Consider possible bugs in the system or test infrastructure (e.g. [races in
+glibc](https://bugs.chromium.org/p/chromium/issues/detail?id=1010318)).
For browsertest flakes, consider possible inter-process issues, such as the
-renderer taking too long or returning something unexpected.
+renderer taking too long or returning something unexpected (e.g. [flaky
+RenderFrameHostImplBrowserTest](https://bugs.chromium.org/p/chromium/issues/detail?id=1120305)).
->TODO: Add more tips for common flake causes
+For browsertest flakes that check EvalJs results, make sure test objects are not
+destroyed before JS may read their values (e.g. [flaky
+PaymentAppBrowserTest](https://chromium.googlesource.com/chromium/src/+/6089f3480c5036c73464661b3b1b6b82807b56a3)).
## Preventing similar flakes
diff --git a/chromium/docs/testing/regression-test-selection.md b/chromium/docs/testing/regression-test-selection.md
new file mode 100644
index 00000000000..1877a063908
--- /dev/null
+++ b/chromium/docs/testing/regression-test-selection.md
@@ -0,0 +1,11 @@
+# Regression test selection (RTS)
+
+Regression Test Selection (RTS) is a technique to intellegently select tests to
+run, without spending too many resources on testing, but still detecting bad
+code changes.
+
+[TOC]
+
+## Design Doc
+
+- [Browser RTS](bit.ly/chromium-rts)
diff --git a/chromium/docs/testing/testing_in_chromium.md b/chromium/docs/testing/testing_in_chromium.md
index b7e97bed0f5..33e2868710b 100644
--- a/chromium/docs/testing/testing_in_chromium.md
+++ b/chromium/docs/testing/testing_in_chromium.md
@@ -160,7 +160,7 @@ development pain for other and then fix it later. "[How do I disable a flaky
test]" has instructions on how to disable a flaky test.
[gtest]: https://github.com/google/googletest
-[Simple gtests]: https://github.com/google/googletest/blob/master/googletest/docs/primer.md#simple-tests
+[Simple gtests]: https://github.com/google/googletest/blob/master/docs/primer.md#simple-tests
[Junit]: https://developer.android.com/training/testing/junit-rules
[Instrumentation Tests]: https://chromium.googlesource.com/chromium/src/+/master/testing/android/docs/instrumentation.md
[EarlGrey]: https://github.com/google/EarlGrey
diff --git a/chromium/docs/testing/web_platform_tests.md b/chromium/docs/testing/web_platform_tests.md
index eb166a1fc1b..0abf3ec2ae1 100644
--- a/chromium/docs/testing/web_platform_tests.md
+++ b/chromium/docs/testing/web_platform_tests.md
@@ -74,13 +74,13 @@ allowed in WPT for this purpose. Please reach out to
ecosystem-infra@chromium.org before following the process below for adding a new
test-only API:
- 1. Create a full list of `*.mojom.js` files that you need, including all
- dependencies. `mojo_bindings.js` loads dependencies recursively by default,
+ 1. Create a full list of `*.mojom.m.js` files that you need, including all
+ dependencies. Generated modules load dependencies recursively by default,
so you can check the network panel of DevTools to see the full list of
dependencies it loads.
2. Check [FILES.cfg](../../chrome/tools/build/linux/FILES.cfg) and add any
- missing `*.mojom.js` files to the `mojojs.zip` archive. Globs are supported
- in `filename`. Do not copy Mojom bindings into WPT.
+ missing `*.mojom.m.js` files to the `mojojs.zip` archive. Globs are
+ supported in `filename`. Do not copy Mojom bindings into WPT.
3. Meanwhile in Chromium, you can create a helper for your WPT tests to do
browser-specific setup using
[test-only-api.js](../../third_party/blink/web_tests/external/wpt/resources/test-only-api.js).
@@ -139,8 +139,9 @@ Please see the `wpt_internal`
**Note**: A significant downside of `wpt_internal` is that your tests may be
broken by upstream changes to the resources scripts (e.g. `testharness.js`), as
`wpt_internal` does not use the forked version of `testharness.js` used by all
-other non-`external/wpt` tests. Use of [WPT-NOTIFY](#wpt_notify) is recommended
-to ensure you are notified of breakages.
+other non-`external/wpt` tests. Use of [new failure
+notifications](#new-failure-notifications) is recommended to ensure you are
+notified of breakages.
## Running tests
@@ -200,19 +201,24 @@ For maintainers:
- If the importer starts misbehaving, it can be disabled by landing a
[CL to skip the update step](https://crrev.com/c/1961906/).
-### WPT-NOTIFY
+### New failure notifications
Test owners can elect to have the importer automatically file bugs against a
component when imported changes introduce failures. This includes new tests that
fail in Chromium, as well as new failures introduced to an existing test. To
-opt-in to this functionality, create an `OWNERS` file in the appropriate
-`external/wpt/` subdirectory that contains the `WPT-NOTIFY` tag. For example,
-`external/wpt/css/css-grid/OWNERS` looks like:
+opt-in to this functionality, create an `DIR_METADATA` file in the appropriate
+`external/wpt/` subdirectory that contains at least `wpt.notify` and
+`monorail.component` fields. For example, `external/wpt/css/css-grid/DIR_METADATA`
+looks like:
```
-# TEAM: layout-dev@chromium.org
-# COMPONENT: Blink>Layout>Grid
-# WPT-NOTIFY: true
+monorail {
+ component: "Blink>Layout>Grid"
+}
+team_email: "layout-dev@chromium.org"
+wpt {
+ notify: YES
+}
```
When a test under `external/wpt/css/css-grid/` newly fails in a WPT import, the
@@ -220,8 +226,8 @@ importer will automatically file a bug against the Blink>Layout>Grid component
in [crbug.com][https://crbug.com], with details of which test failed and the
output.
-Note that we are considering making WPT-NOTIFY opt-out instead of opt-in: see
-https://crbug.com/845232
+Note that we are considering making the notifications opt-out instead of
+opt-in: see https://crbug.com/845232
### Skipped tests (and how to re-enable them)
diff --git a/chromium/docs/testing/web_platform_tests_wptrunner.md b/chromium/docs/testing/web_platform_tests_wptrunner.md
index 54120ac8a70..2150f759f78 100644
--- a/chromium/docs/testing/web_platform_tests_wptrunner.md
+++ b/chromium/docs/testing/web_platform_tests_wptrunner.md
@@ -69,14 +69,14 @@ have been brought up:
* [linux-wpt-identity-fyi-rel](https://ci.chromium.org/p/chromium/builders/ci/linux-wpt-identity-fyi-rel),
which runs tests under `external/wpt/webauthn/`.
-* [linux-wpt-payments-fyi-rel](https://ci.chromium.org/p/chromium/builders/ci/linux-wpt-payments-fyi-rel),
- which runs tests under `external/wpt/payment-{handler, method-basic-card,
- request}`.
+* [linux-wpt-input-fyi-rel](https://ci.chromium.org/p/chromium/builders/ci/linux-wpt-input-fyi-rel),
+ which runs tests under `external/wpt/{input-events, pointerevents, uievents}`,
+ as well as `external/wpt/infrastructure/testdriver/actions/`
These bots run on the waterfall, but can also be run on CLs by clicking the
`Choose Tryjobs` button in Gerrit followed by searching for the bot name in the
modal dialog that appears. One can also include the tag `Cq-Include-Trybots:
-luci.chromium.try:linux-wpt-identity-fyi-rel` (or payments) in the description
+luci.chromium.try:linux-wpt-identity-fyi-rel` (or input) in the description
for the CL, which will make the bot mandatory for that CL.
Results for the bots use the existing layout test results viewer
diff --git a/chromium/docs/testing/web_test_expectations.md b/chromium/docs/testing/web_test_expectations.md
index ea0fb232e72..bc37e1faa30 100644
--- a/chromium/docs/testing/web_test_expectations.md
+++ b/chromium/docs/testing/web_test_expectations.md
@@ -102,11 +102,17 @@ results from try jobs, by using the command-tool
1. First, upload a CL.
2. Trigger try jobs by running `blink_tool.py rebaseline-cl`. This should
trigger jobs on
- [tryserver.blink](https://build.chromium.org/p/tryserver.blink/builders).
+ [tryserver.blink](https://ci.chromium.org/p/chromium/g/tryserver.blink/builders).
+ In addition, this will also trigger the CQ try builders that run blink web tests.
+ linux-rel, mac-rel and win10_chromium_x64_rel_ng.
+ Optionally one can choose to trigger only blink try bots alone.
+ Run the tool with the option -
+ `blink_tool.py rebaseline-cl --use-blink-try-bots-only`
3. Wait for all try jobs to finish.
4. Run `blink_tool.py rebaseline-cl` again to fetch new baselines.
By default, this will download new baselines for any failing tests
- in the try jobs.
+ in the blink try jobs and CQ try bots.
+ Again, there is an option to use only blink try jobs results for rebaselining.
(Run `blink_tool.py rebaseline-cl --help` for more specific options.)
5. Commit the new baselines and upload a new patch.
@@ -160,7 +166,7 @@ details.
longer than the usual timeout to run. Slow tests are given 5x the usual
timeout.
* [SmokeTests](../../third_party/blink/web_tests/SmokeTests): A small subset
- of tests that we run on the Android bot.
+ of tests that we run on the Fuchsia bots.
* [StaleTestExpectations](../../third_party/blink/web_tests/StaleTestExpectations):
Platform-specific lines that have been in TestExpectations for many months.
They're moved here to get them out of the way of people doing rebaselines
diff --git a/chromium/docs/testing/web_tests.md b/chromium/docs/testing/web_tests.md
index 33963609749..168155a44fb 100644
--- a/chromium/docs/testing/web_tests.md
+++ b/chromium/docs/testing/web_tests.md
@@ -23,6 +23,15 @@ Chrome-specific tests only.
## Running Web Tests
+### Supported Platforms
+
+* Linux
+* MacOS
+* Windows
+* Fuchsia
+
+Android is [not supported](https://crbug.com/567947).
+
### Initial Setup
Before you can run the web tests, you need to build the `blink_tests` target
@@ -32,16 +41,6 @@ to get `content_shell` and all of the other needed binaries.
autoninja -C out/Default blink_tests
```
-On **Android** (web test support
-[currently limited to KitKat and earlier](https://crbug.com/567947)) you need to
-build and install `content_shell_apk` instead. See also:
-[Android Build Instructions](../android_build_instructions.md).
-
-```bash
-autoninja -C out/Default content_shell_apk
-adb install -r out/Default/apks/ContentShell.apk
-```
-
On **Mac**, you probably want to strip the content_shell binary before starting
the tests. If you don't, you'll have 5-10 running concurrently, all stuck being
examined by the OS crash reporter. This may cause other failures like timeouts
@@ -53,8 +52,6 @@ strip ./xcodebuild/{Debug,Release}/content_shell.app/Contents/MacOS/content_shel
### Running the Tests
-TODO: mention `testing/xvfb.py`
-
The test runner script is in `third_party/blink/tools/run_web_tests.py`.
To specify which build directory to use (e.g. out/Default, out/Release,
@@ -65,14 +62,10 @@ use the build in `out/Default`, use:
third_party/blink/tools/run_web_tests.py -t Default
```
-For Android (if your build directory is `out/android`):
-
-```bash
-third_party/blink/tools/run_web_tests.py -t android --android
-```
-
*** promo
-Windows users need to use `third_party/blink/tools/run_web_tests.bat` instead.
+* Windows users need to use `third_party/blink/tools/run_web_tests.bat` instead.
+* Linux users should not use `testing/xvfb.py`; `run_web_tests.py` manages Xvfb
+ itself.
***
Tests marked as `[ Skip ]` in
@@ -86,12 +79,11 @@ learn more about TestExpectations and related files.
*** promo
Currently only the tests listed in
-[SmokeTests](../../third_party/blink/web_tests/SmokeTests)
-are run on the Android bots, since running all web tests takes too long on
-Android (and may still have some infrastructure issues). Most developers focus
-their Blink testing on Linux. We rely on the fact that the Linux and Android
-behavior is nearly identical for scenarios outside those covered by the smoke
-tests.
+[SmokeTests](../../third_party/blink/web_tests/SmokeTests) are run on the
+Fuchsia bots, since running all web tests takes too long on Fuchshia. Most
+developers focus their Blink testing on Linux. We rely on the fact that the
+Linux and Fuchsia behavior is nearly identical for scenarios outside those
+covered by the smoke tests.
***
To run only some of the tests, specify their directories or filenames as
diff --git a/chromium/docs/testing/web_tests_in_content_shell.md b/chromium/docs/testing/web_tests_in_content_shell.md
index 3de16d97a8c..d806f410420 100644
--- a/chromium/docs/testing/web_tests_in_content_shell.md
+++ b/chromium/docs/testing/web_tests_in_content_shell.md
@@ -1,5 +1,7 @@
# Running web tests using the content shell
+[TOC]
+
## Compiling
If you want to run web tests,
@@ -64,7 +66,7 @@ browser](#As-a-simple-browser).
In rare cases, to run Content Shell in the exact same way as
`run_web_tests.py` runs it, you need to run it in the
-[protocol mode](../../content/shell/browser/web_test/test_info_extractor.h).
+[protocol mode](../../content/web_test/browser/test_info_extractor.h).
*** note
On the Mac, use `Content Shell.app`, not `content_shell`.