summaryrefslogtreecommitdiff
path: root/render-test
Commit message (Collapse)AuthorAgeFilesLines
* [core] Modularize FileSource codebase (#15768)Alexander Shalamov2020-01-135-22/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * [core] Introduce FileSourceManager and use it for default platform impl - Add `FileSourceManager` interface that provides access to `FileSource` instances and means of registering / unregistering `FileSource` factories - Split `DefaultFileSource` into smaller parts - Add `DatabaseFileSource` interface and it's default implementation - Remove inter-dependencies between concrete `FileSource` classes * [build] Add files to next build system * [core] Add generic property setters / getters * [core] Remove setOnlineStatus from OnlineFileSource interface * [core] Hide threading implementation details from DatabaseFileSource interface * [core] Make DB file source methods virtual * [core] Add documentation for DatabaseFileSource and rename one method * [core] Use simple callback instead of ActorRef * [core] Remove ActorRef from OnlineFileSource public header * [core] Add callback to FileSource::forward async API * [core] Pass OfflineRegionDefinition by value * [core] Update tests to use modular file sources * [core] Update unit tests * [core] Update unit tests after rebase * [core] Backport low prio fix for cached requests * [core] Backport pack database * [core] Return removed factory from unRegisterFileSourceFactory * [core] Rename shadowed args in onlinefilesource * [core] Remove simple std::function callback aliases * [core] Expose online file source property keys in public header file * [test-runner] Add proxy file source test runner * [cache] Update mbgl-cache utility to use new file source * [metrics] Rebaseline binary size metrics * [offline] Update offline utility * [core] Update changelog
* [render-test] Make the manifest mandatoryThiago Marcos P. Santos2020-01-082-8/+6
| | | | At the moment we have an invalid path as default.
* [render-test] Remove positional argumentThiago Marcos P. Santos2020-01-083-43/+20
| | | | Not in use, for running a single test, use -f instead.
* [core] Make it possible to reopen the database read-only for testingThiago Marcos P. Santos2020-01-081-0/+6
| | | | | | Needed for Android, because the database is in readonly filesystem and also because it is annoying to see files modified every time we run the tests.
* [metrics] Use the cache database instead of file paths for fixturesThiago Marcos P. Santos2020-01-081-12/+2
| | | | | Now that we use a cache, we no longer need "asset_path" and "vendor_path".
* [render-test] Use image from the cache for addImage operationThiago Marcos P. Santos2020-01-084-11/+37
| | | | Instead of trying to read it from the filesystem.
* [render-test] Effectively use the offline cache for fixturesThiago Marcos P. Santos2020-01-089-267/+87
|
* [render-test] Add access token to the test manifestThiago Marcos P. Santos2020-01-082-1/+27
| | | | | Needed if we want to run tests online. Will default to the MAPBOX_ACCESS_TOKEN environment value.
* [render-test] Add flag to run tests onlineThiago Marcos P. Santos2020-01-081-3/+11
| | | | | And also make it not go to the network by setting the status explicitly when when the -o flag is not passed.
* [render-test] Add cache database path to the manifestThiago Marcos P. Santos2020-01-082-0/+18
| | | | A path relative to the manifest pointing at the data cache.
* [render-test] Fixed typo and remove unusedThiago Marcos P. Santos2020-01-081-29/+14
|
* [test runner] Show all tiles independently in Tile modeMikhail Pozdnyakov2019-12-181-7/+44
|
* [test runner] Rendering a single tile in Tile modeMikhail Pozdnyakov2019-12-183-11/+40
|
* [render-tests] Fix tests not rendering when both metrics and render failsThiago Marcos P. Santos2019-12-171-5/+7
|
* [render-test] Cleanup on the render tests resultsThiago Marcos P. Santos2019-12-165-120/+129
| | | | | Simplify the code path and make it print both rendering error and metrics on fail.
* [build] Collect and publish new baselinesThiago Marcos P. Santos2019-12-161-1/+1
| | | | Also fail the bot in case there are new baselines.
* [build] Reorganize the render testsThiago Marcos P. Santos2019-12-16100-1216/+5
| | | | | | Move everything that might collect metrics to the metrics/ directory, the only test manifests inside the render-test/ directory are the unit tests of the probes.
* [render-test] Fix metrics passing when failingThiago Marcos P. Santos2019-12-163-16/+32
| | | | | | Metrics will now fail, but the tool will return 0 when *ONLY* metrics fail, to make the bot continue. We will also run ignored tests and check if they are ignored but passing.
* [test runner] Explicitly initialize TestContext POD membersMikhail Pozdnyakov2019-12-141-2/+2
|
* [test-runner] Fix network probeAlexander Shalamov2019-12-131-3/+7
|
* [render test] Fix probes failure reportsMikhail Pozdnyakov2019-12-133-11/+7
|
* [test runner] Refactor injected probes operations codeMikhail Pozdnyakov2019-12-132-126/+130
|
* [test runner] Split operations parsing and executionMikhail Pozdnyakov2019-12-135-663/+805
|
* [test runner] Introduce TestOperationMikhail Pozdnyakov2019-12-132-9/+18
|
* [test runner] Introduce generic TestContextMikhail Pozdnyakov2019-12-132-59/+80
|
* [test-runner] Don't update nor create metrics baseline for ignored testAlexander Shalamov2019-12-123-2/+5
|
* [test-runner] Fix -Werror=shadow issues in test runnerAlexander Shalamov2019-12-121-19/+18
|
* [metrics] Run Android RenderTestRunner bot in rebaseline mode (#16036)zmiao2019-12-111146-53552/+6
|
* [test-runner] Add desctiption for --update commandAlexander Shalamov2019-12-101-1/+4
|
* [test runner] Report absolute path to the results html fileMikhail Pozdnyakov2019-12-101-1/+1
|
* [test-runner] Add rebaseline flagAlexander Shalamov2019-12-093-7/+24
| | | | | Rebaseline flag allows to update or create new metric, in cases when metric check is failed or expected metric is missing.
* [test-runner] Simplify update option and remove obsolete ifdefsAlexander Shalamov2019-12-092-33/+12
|
* [metrics] Create gcc8-release metrics configurationAlexander Shalamov2019-12-091145-40096/+0
|
* [metrics] Update render-test-probe-unit-tests memory expectationsMikhail Pozdnyakov2019-12-094-16/+16
|
* [core] Refactoring transform state (#15956)zmiao2019-12-052-4/+4
| | | | | | | | | | * [core] Refactoring transform state class * [core] update matrix in transform, Fix precision * [core] Make matrix mutable so that we only update them when needed * [core] Add getters for matrices
* [test-runner] Make Android RenderTestRunner bot fail when test cases are not ↵zmiao2019-12-0428-14/+342
| | | | | | | | | | | | | | fully passed (#15994) * [test-runner] Fix failed tests, generate android specific expectations * [test-runner] Check render test runner testing result * [test-runner] Add error case to android ignore list * [test-runner] Update metrics * [test-runner] Move Android render tests with big diff(>0.006) to ignore list
* [test-runner] Allow to define filter inside manifest fileAlexander Shalamov2019-12-035-7/+17
| | | | | Command line option is still available and allows to override filter that is defined in a manifest file.
* [test runner] Update test commands for Layer::setProperty()Mikhail Pozdnyakov2019-12-031-20/+4
|
* [metrics] Add baselines for formatted-images and line-dasharray render testsAlexander Shalamov2019-12-029-0/+315
|
* [test-runner] Add android RenderTestRunner baseline metrics (#15979)zmiao2019-12-021136-3/+53232
| | | | | | | | | | * [test-runner] Enable probe-testing android render test runner * [test-runner] Add pixel4 baseline metrics * Add git ignored metrics * Make renderTestrunner run two times with different manifest config
* [test-runner] Log all 'probeFileSize' failuresAlexander Shalamov2019-12-021-5/+8
| | | | | If there are multiple file size probes in a test file, it is better to report all failures instead of first failed probe.
* [test-runner] Run render tests with metrics on linuxJuha Alanen2019-11-294-1/+25
|
* [test-runner] Add baseline metrics for linuxJuha Alanen2019-11-291134-0/+39762
|
* [render-test] Add support for measuring compressed file sizeThiago Marcos P. Santos2019-11-283-4/+30
| | | | | Useful because for platform such as Android, we want to measure the size of the library inside the APK, which is usually compressed.
* [metrics] Make the parsing of metrics even more strictThiago Marcos P. Santos2019-11-281-1/+1
| | | | File must be `metrics.json`.
* [render-test] Fix regex matching the directory nameThiago Marcos P. Santos2019-11-271-3/+3
| | | | | This code was matching `style.json` instead of `metrics.json` if the `style.json` was contained inside `metrics/.../style.json`.
* [test runner] Enable test results update from cmd lineMikhail Pozdnyakov2019-11-273-10/+45
| | | | | | | | Added the following command line argument: ``` -u[update], --update=[update] Test results update mode. Supported values are: "default", "platform", "metrics" ```
* [test-runner] Move probe related command line args to manifestAlexander Shalamov2019-11-275-38/+48
| | | | | | | | | | | - Move probe related command line arguments to manifest, example: "probes": ["probeGFX", "probeNetwork", "probeMemory"], "metric_path": "metrics/linux" - Use manifest file name as a postfix for a result page - Move injected probes 'begin' section before map object creation - Generalize artifact storing shell script
* [render-test] Add metric_paths into Manifest (#15973)zmiao2019-11-2711-197/+275
| | | | | | | | | | * [render-test]Add expected metrics path in manifest file * seperate metrics.json and expected.png * fix expected path parsing * keep expectation path if update is enabled
* [test-runner] Add render test runner options for injecting probesAlexander Shalamov2019-11-223-48/+177
| | | | | This allows injection of memory, gfx and network probes before and after operations defined in render test are run.
View Lorry Controller Status