summaryrefslogtreecommitdiff
path: root/chromium/docs/testing
diff options
context:
space:
mode:
authorAllan Sandfeld Jensen <allan.jensen@qt.io>2020-07-16 11:45:35 +0200
committerAllan Sandfeld Jensen <allan.jensen@qt.io>2020-07-17 08:59:23 +0000
commit552906b0f222c5d5dd11b9fd73829d510980461a (patch)
tree3a11e6ed0538a81dd83b20cf3a4783e297f26d91 /chromium/docs/testing
parent1b05827804eaf047779b597718c03e7d38344261 (diff)
downloadqtwebengine-chromium-552906b0f222c5d5dd11b9fd73829d510980461a.tar.gz
BASELINE: Update Chromium to 83.0.4103.122
Change-Id: Ie3a82f5bb0076eec2a7c6a6162326b4301ee291e Reviewed-by: Michael BrĂ¼ning <michael.bruning@qt.io>
Diffstat (limited to 'chromium/docs/testing')
-rw-r--r--chromium/docs/testing/android_test_instructions.md2
-rw-r--r--chromium/docs/testing/code_coverage.md3
-rw-r--r--chromium/docs/testing/code_coverage_in_gerrit.md2
-rw-r--r--chromium/docs/testing/rendering_representative_perf_tests.md33
-rw-r--r--chromium/docs/testing/test_browser_dialog.md5
-rw-r--r--chromium/docs/testing/using_crashpad_with_content_shell.md14
-rw-r--r--chromium/docs/testing/web_platform_tests.md4
-rw-r--r--chromium/docs/testing/web_test_expectations.md14
-rw-r--r--chromium/docs/testing/web_tests.md11
-rw-r--r--chromium/docs/testing/web_tests_in_content_shell.md2
-rw-r--r--chromium/docs/testing/web_tests_linux.md4
-rw-r--r--chromium/docs/testing/writing_js_unit_tests.md2
12 files changed, 61 insertions, 35 deletions
diff --git a/chromium/docs/testing/android_test_instructions.md b/chromium/docs/testing/android_test_instructions.md
index b7280759691..ec2786c0c04 100644
--- a/chromium/docs/testing/android_test_instructions.md
+++ b/chromium/docs/testing/android_test_instructions.md
@@ -265,7 +265,7 @@ out/Debug/bin/run_content_shell_test_apk --wait-for-java-debugger
If running with `is_debug=false`, Java stacks from logcat need to be fixed up:
```shell
-out/Release/bin/java_deobfuscate out/Release/apks/ChromePublicTest.apk.mapping < stacktrace.txt
+build/android/stacktrace/java_deobfuscate.py out/Release/apks/ChromePublicTest.apk.mapping < stacktrace.txt
```
Any stacks produced by test runner output will already be deobfuscated.
diff --git a/chromium/docs/testing/code_coverage.md b/chromium/docs/testing/code_coverage.md
index b437d132770..b3c9fff93c9 100644
--- a/chromium/docs/testing/code_coverage.md
+++ b/chromium/docs/testing/code_coverage.md
@@ -130,7 +130,8 @@ Here is an example usage:
```
$ gn gen out/coverage \
- --args='use_clang_coverage=true is_component_build=false dcheck_always_on=true'
+ --args="use_clang_coverage=true is_component_build=false
+ dcheck_always_on=true is_debug=false"
$ python tools/code_coverage/coverage.py \
crypto_unittests url_unittests \
-b out/coverage -o out/report \
diff --git a/chromium/docs/testing/code_coverage_in_gerrit.md b/chromium/docs/testing/code_coverage_in_gerrit.md
index b0f102518bc..44767b64e91 100644
--- a/chromium/docs/testing/code_coverage_in_gerrit.md
+++ b/chromium/docs/testing/code_coverage_in_gerrit.md
@@ -70,4 +70,4 @@ in Gerrit.
[clang_code_coverage_wrapper]: clang_code_coverage_wrapper.md
[chromium-coverage Gerrit plugin]: https://chromium.googlesource.com/infra/gerrit-plugins/code-coverage/
[Chromium on Chromium OS]: https://chromium.googlesource.com/chromium/src/+/master/docs/chromeos_build_instructions.md
-[Chromium on Linux]: https://chromium.googlesource.com/chromium/src/+/master/docs/linux_build_instructions.md
+[Chromium on Linux]: https://chromium.googlesource.com/chromium/src/+/master/docs/linux/build_instructions.md
diff --git a/chromium/docs/testing/rendering_representative_perf_tests.md b/chromium/docs/testing/rendering_representative_perf_tests.md
index da58248b552..c4b8c5a4b9f 100644
--- a/chromium/docs/testing/rendering_representative_perf_tests.md
+++ b/chromium/docs/testing/rendering_representative_perf_tests.md
@@ -8,7 +8,26 @@ compared with the historical upper limit described in [`src/testing/scripts/repr
[TOC]
-## Clustering the Benchmark and Choosing Representatives
+## Investigating Representative perf test failures
+
+The representative perf tests runs a set of stories and measures the `frame_times` metric during the interaction to determine performance.
+The common reasons of failure are (1) the measured value is higher than the expected value and (2) crashes while running the tests.
+
+In the case of difference between values, a message would be logged in the output of the test explaining so.
+Example:`animometer_webgl_attrib_arrays higher average frame_times(21.095) compared to upper limit (17.062)` ([Link to the task](https://chromium-swarm.appspot.com/task?d=true&id=4a92ea7a6b40d010))
+This means that the animometer_webgl_attrib_arrays story has the average frame_times of 21 ms and the recorded upper limit for the story (in the tested platform) is 17 ms.
+
+In these cases the failed story will be ran three more times to make sure that this has not been a flake, and the new result (average of three runs) will be reported in the same format.
+For deeper investigation of such cases you can find the traces of the runs in the isolated outputs of the test. In the isolated outputs directory look at output.json for the initial run and at re_run_failures/output.json for the three re-runs.
+
+In the `output.json` file, you can find the name of the story and under the trace.html field of the story a gs:// link to the trace ([Example](https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=f8961d773cdf0bf121525f29024c0e6c19d42e61&as=output.json)).
+To download the trace run: `gsutil cp gs://link_from_output.json trace_name.html`
+
+Also if tests fail with no specific messages in the output, it will be useful to check the {benchmark}/benchmark_log.txt file in the isolated outputs directory for more detailed log of the failure.
+
+## Maintaining Representative Performance Tests
+
+### Clustering the Benchmark and Choosing Representatives
The clustering of the benchmark is based on the historical values recorded for
`frame_times`. For steps on clustering the benchmark check [Clustering benchmark stories](../../tools/perf/experimental/story_clustering/README.md).
@@ -21,7 +40,7 @@ Currently there are three sets of representatives described by story tags below:
Adding more stories to representatives or removing stories from the set is
managed by adding and removing story tags above to stories in [rendering benchmark](../../tools/perf/page_sets/rendering).
-## Updating the Upper Limits
+### Updating the Upper Limits
The upper limits for averages and confidence interval (CI) ranges of
`frame_times` described in [`src/testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json`](../../testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json)
@@ -37,8 +56,14 @@ Updating these values can be done by running [`src/tools/perf/experimental/repre
The script will create a new JSON file using the values of recent runs in place
of [`src/testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json`](../../testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json).
-## Updating Expectations
+### Updating Expectations
To skip any of the tests, update
[`src/tools/perf/expectations.config`](../../tools/perf/expectations.config) and
-add the story under rendering benchmark. \ No newline at end of file
+add the story under rendering benchmark (Examples [1](https://chromium-review.googlesource.com/c/chromium/src/+/2055681), [2](https://chromium-review.googlesource.com/c/chromium/src/+/1901357)).
+This expectations file disables the story on the rendering benchmark, which rendering_representative_perf_tests are part of.
+So please add the a bug for each skipped test and link to `Internals>GPU>Metrics`.
+
+If the test is part of representative perf tests on Windows or MacOS, this
+should be done under rendering.desktop benchmark and if it's a test on Android
+under rendering.mobile. \ No newline at end of file
diff --git a/chromium/docs/testing/test_browser_dialog.md b/chromium/docs/testing/test_browser_dialog.md
index e602652f8d9..eda1b9ccaf6 100644
--- a/chromium/docs/testing/test_browser_dialog.md
+++ b/chromium/docs/testing/test_browser_dialog.md
@@ -175,9 +175,8 @@ Even discovering the full set of UI present for each platform in Chrome is
* An alternative is to maintain a working end-to-end build target executable
to do this, but this has additional costs (and is hard).
- * E.g. setup/teardown of low-level functions (
- `InitializeGLOneOffPlatform()`,
- `MaterialDesignController::Initialize()`, etc.).
+ * E.g. setup/teardown of low-level functions
+ (`InitializeGLOneOffPlatform()`, etc.).
* Why not chrome.exe?
* E.g. a scrappy chrome:// page with links to invoke UI would be great!
diff --git a/chromium/docs/testing/using_crashpad_with_content_shell.md b/chromium/docs/testing/using_crashpad_with_content_shell.md
index 4c08a9ec17d..5152592a088 100644
--- a/chromium/docs/testing/using_crashpad_with_content_shell.md
+++ b/chromium/docs/testing/using_crashpad_with_content_shell.md
@@ -16,12 +16,6 @@ building. This slows down linking several minutes, so don't just always set it
by default.
***
-*** note
-**Linux:** Add `use_debug_fission = true` to your [gn build
-arguments](https://gn.googlesource.com/gn/+/master/docs/quick_start.md) before
-building.
-***
-
Then, create a directory where the crash dumps will be stored:
* Linux/Mac:
@@ -67,13 +61,9 @@ Crashpad can be enabled by passing `--enable-crash-reporter` and
## Retrieving the crash dump
-On Linux and Android, we first have to retrieve the crash dump. On Mac and
-Windows, this step can be skipped.
+On Android, we first have to retrieve the crash dump. On other platforms, this
+step can be skipped.
-* Linux:
- ```bash
- components/crash/content/tools/dmp2minidump.py /tmp/crashes/*.dmp /tmp/minidump
- ```
* Android:
```bash
adb pull $(adb shell ls /data/data/org.chromium.content_shell_apk/cache/pending/*.dmp) /tmp/chromium-renderer-minidump.dmp
diff --git a/chromium/docs/testing/web_platform_tests.md b/chromium/docs/testing/web_platform_tests.md
index 16d866d8afa..ddff2089fa2 100644
--- a/chromium/docs/testing/web_platform_tests.md
+++ b/chromium/docs/testing/web_platform_tests.md
@@ -48,8 +48,8 @@ For maintainers:
- The source lives in
[third_party/blink/tools/wpt_import.py](../../third_party/blink/tools/wpt_import.py).
-- If the importer starts misbehaving, it could be disabled by turning off the
- auto-import mode by landing [this CL](https://crrev.com/c/617479/).
+- If the importer starts misbehaving, it can be disabled by landing a
+ [CL to skip the update step](https://crrev.com/c/1961906/).
### Failures caused by automatic imports.
diff --git a/chromium/docs/testing/web_test_expectations.md b/chromium/docs/testing/web_test_expectations.md
index d101278db52..ea0fb232e72 100644
--- a/chromium/docs/testing/web_test_expectations.md
+++ b/chromium/docs/testing/web_test_expectations.md
@@ -116,8 +116,6 @@ is no period of time when the web test results are ignored.
#### Options
-### Rebaselining with try jobs
-
The tests which `blink_tool.py rebaseline-cl` tries to download new baselines for
depends on its arguments.
@@ -130,6 +128,18 @@ depends on its arguments.
* If some of the try jobs failed to run, and you wish to continue rebaselining
assuming that there are no platform-specific results for those platforms,
you can add the flag `--fill-missing`.
+* By default, it finds the try jobs by looking at the latest patchset. If you
+ have finished try jobs that are associated with an earlier patchset and you
+ want to use them instead of scheduling new try jobs, you can add the flag
+ `--patchset=n` to specify the patchset. This is very useful when the CL has
+ 'trivial' patchsets that are created e.g. by editing the CL descrpition.
+
+### Local manual rebaselining
+
+This is often useful for rebaselining flag-specific results, or rebaselining
+just for the local platform.
+See [Rebaselining Web Tests](./web_tests.md#Rebaselining-Web-Tests) for more
+details.
## Kinds of expectations files
diff --git a/chromium/docs/testing/web_tests.md b/chromium/docs/testing/web_tests.md
index 3ab8de8c3ba..1201c8f89c4 100644
--- a/chromium/docs/testing/web_tests.md
+++ b/chromium/docs/testing/web_tests.md
@@ -13,6 +13,12 @@ Note that we changed the term "layout tests" to "web tests".
Please assume these terms mean the identical stuff. We also call it as
"WebKit tests" and "WebKit layout tests".
+["Web platform tests"](./web_platform_tests.md) (WPT) are the preferred form of
+web tests and are located at
+[web_tests/external/wpt](/third_party/blink/web_tests/external/wpt).
+Tests that should work across browsers go there. Other directories are for
+Chrome-specific tests only.
+
[TOC]
## Running Web Tests
@@ -581,11 +587,6 @@ files. You can follow the steps below for easier review.
3. Request review of the CL and tell the reviewer to compare the patch sets that
were uploaded in step 1 and step 2 to see the differences of the rebaselines.
-## web-platform-tests
-
-In addition to web tests developed and run just by the Blink team, there is
-also a shared test suite, see [web-platform-tests](./web_platform_tests.md).
-
## Known Issues
See
diff --git a/chromium/docs/testing/web_tests_in_content_shell.md b/chromium/docs/testing/web_tests_in_content_shell.md
index 609bb5d7677..9b4c4229481 100644
--- a/chromium/docs/testing/web_tests_in_content_shell.md
+++ b/chromium/docs/testing/web_tests_in_content_shell.md
@@ -60,7 +60,7 @@ browser](#As-a-simple-browser).
In rare cases, to run Content Shell in the exact same way as
`run_web_tests.py` runs it, you need to run it in the
-[protocol mode](../../content_shell/browser/web_test/test_info_extractor.h).
+[protocol mode](../../content/shell/browser/web_test/test_info_extractor.h).
*** note
On the Mac, use `Content Shell.app`, not `content_shell`.
diff --git a/chromium/docs/testing/web_tests_linux.md b/chromium/docs/testing/web_tests_linux.md
index 96ede1d937d..b9b94b1d673 100644
--- a/chromium/docs/testing/web_tests_linux.md
+++ b/chromium/docs/testing/web_tests_linux.md
@@ -1,6 +1,6 @@
# Running web tests on Linux
-1. Build `blink_tests` (see [Linux-specific build instructions](https://chromium.googlesource.com/chromium/src/+/master/docs/linux_build_instructions.md))
+1. Build `blink_tests` (see [Linux-specific build instructions](https://chromium.googlesource.com/chromium/src/+/master/docs/linux/build_instructions.md))
1. Checkout the web tests
* If you have an entry in your `.gclient` file that includes
"web_tests", you may need to comment it out and sync.
@@ -59,7 +59,7 @@ If `fast/dom/object-plugin-hides-properties.html` and
using a chroot jail with Lucid 64-bit user space to be sure that your system
matches the checked in baselines. You can use `build/install-chroot.sh` to
set up a Lucid 64 chroot. Learn more about
- [using a linux chroot](using_a_linux_chroot.md).
+ [using a linux chroot](linux/using_a_chroot.md).
## Getting a web test into a debugger
diff --git a/chromium/docs/testing/writing_js_unit_tests.md b/chromium/docs/testing/writing_js_unit_tests.md
index 522f7541027..68e5346f229 100644
--- a/chromium/docs/testing/writing_js_unit_tests.md
+++ b/chromium/docs/testing/writing_js_unit_tests.md
@@ -366,7 +366,7 @@ examples. Here are some useful examples:
[oobe_webui_browsertest.js](https://cs.chromium.org/chromium/src/chrome/test/data/chromeos/oobe_webui_browsertest.js) -
makes the browser fullscreen before running the JS test.
-[saml_password_attributes_test.unitjs](https://cs.chromium.org/chromium/src/chrome/browser/resources/chromeos/login/saml_password_attributes_test.unitjs) -
+[saml_password_attributes_test.unitjs](https://cs.chromium.org/chromium/src/chrome/browser/resources/gaia_auth_host/saml_password_attributes_test.unitjs) -
loads an XML file into a JS global variable before running the JS test.
Another thing that will make your life a little bit easier is being able to