diff options
author | Tobrun <tobrun.van.nuland@gmail.com> | 2016-08-17 08:53:33 +0200 |
---|---|---|
committer | GitHub <noreply@github.com> | 2016-08-17 08:53:33 +0200 |
commit | fff0ebaa1754aabd9584c994624b7b948bec95af (patch) | |
tree | 745006b1a761856e6651738cd7d26e0e08c6cbbf /platform/android/tests | |
parent | 0062cf2f949fe1b0080e2de45aaf0b3d5e028bf3 (diff) | |
download | qtlocation-mapboxgl-fff0ebaa1754aabd9584c994624b7b948bec95af.tar.gz |
5859 generate sanity tests (#5870)
* [android] #5859 - generate espresso tests
[android] #5859 - cleanup xml files, use all the same resource id for map view
[android] #5859 - update scripts
[android] #5859 - update Activity to match mapboxMap naming
[android] #5859 - make SurfaceViewMediaControlActivity compatible
[android] #5859 - make MarkerViewScaleActivity test generation compatible
[android] #5859 - make PolygonActivity test generation compatible
[android] #5859 - integration within make file
[android] #5859 - made CameraPostion generation safe
[android] #5859 - generate sanity test
[android] #5859 - add generated test cases
[android] #5859 - add documentation
* [android] #5859 - remove generated files, introduce gen package, added gitignore to package
* [android] #5859 - regenerated test code
* [android] #5859 - added AWS gradle plugin + bitrise configuration
* [android] #5859 - fixes running locally without having the properties configured
* [android] #6010 - remove python from android sdk
* [android] #5859 - update docs about aws gradle plugin
Diffstat (limited to 'platform/android/tests')
-rw-r--r-- | platform/android/tests/docs/ACTIVITY_SANITY_TEST.md | 17 | ||||
-rw-r--r-- | platform/android/tests/docs/UI_TESTS.md | 89 | ||||
-rw-r--r-- | platform/android/tests/scripts/devicefarm.py | 132 |
3 files changed, 20 insertions, 218 deletions
diff --git a/platform/android/tests/docs/ACTIVITY_SANITY_TEST.md b/platform/android/tests/docs/ACTIVITY_SANITY_TEST.md new file mode 100644 index 0000000000..14e4335111 --- /dev/null +++ b/platform/android/tests/docs/ACTIVITY_SANITY_TEST.md @@ -0,0 +1,17 @@ +# Activity Sanity Tests + +This type of test checks if an Activity doesn't crash at start up. For this project this type of test is usefull since a lot of core crashes occurs at this point and are ideal to test regressions. The only run for 1 sec each so there are fast and fairly reliable. We are currently generating the test stack from every Activity found in the following package: + +> com.mapbox.mapboxsdk.testapp + +The generation is done by executing: + +> make android-generate-test + +This command underneath executes the following js file found in: + +> node scripts/android-generate-test + +and uses the following ejs file as a template. + +>platform/android/MapboxGLAndroidSDKTestApp/src/androidTest/java/com/mapbox/mapboxsdk/activity/activity.junit.ejs diff --git a/platform/android/tests/docs/UI_TESTS.md b/platform/android/tests/docs/UI_TESTS.md index 4d7addea0d..9341cf1e1e 100644 --- a/platform/android/tests/docs/UI_TESTS.md +++ b/platform/android/tests/docs/UI_TESTS.md @@ -59,92 +59,9 @@ You can generate JaCoCo reports from espresso tests by - running the gradle task `createMockDebugCoverageReport` when executing tests. ## Running Espresso test automatically on AWS Device Farm -To automatically execute Espresso tests as part of our CI build, we have created a Python script called [`devicefarm.py`](https://github.com/mapbox/mapbox-gl-native/blob/master/platform/android/tests/scripts/devicefarm.py). - -This script is responsible for: - - uploading an APK + test APK - - scheduling tests - - exiting with a return code - - 0 -> all tests have passed - - 1 otherwise - -### Requirements - - * [Boto 3](http://boto3.readthedocs.org) - * [Requests](http://www.python-requests.org) - -### Running the script - - A sample run would be as follows: - - ``` - $ python devicefarm.py \ - --project-arn "arn:aws:devicefarm:us-west-2:XXXXX" \ - --device-pool-arn "arn:aws:devicefarm:us-west-2::devicepool:YYYYY" \ - --app-apk-path app/build/outputs/apk/app-debug-unaligned.apk \ - --test-apk-path app/build/outputs/apk/app-debug-androidTest-unaligned.apk - ``` - - Where you need to insert your actual project and device ARNs. We follow Boto 3 - conventions to to [set up the AWS credentials](https://github.com/boto/boto3#quick-start). - - You can build the `app-debug-androidTest-unaligned.apk` package with Gradle: - - ``` - ./gradlew assembleAndroidTest - ``` - - To run tests locally, you can use `./gradlew assemble` to build the app APK, and - `./gradlew test --continue` to run unit tests. Finally, `./gradlew connectedAndroidTest` - will run the Espresso tests in a local device. - - A sample output would be as follows: - - ``` - Starting upload: ANDROID_APP - Uploading: ../app/build/outputs/apk/app-debug-unaligned.apk - Checking if the upload succeeded. - Upload not ready (status is INITIALIZED), waiting for 5 seconds. - Starting upload: INSTRUMENTATION_TEST_PACKAGE - Uploading: ../app/build/outputs/apk/app-debug-androidTest-unaligned.apk - Checking if the upload succeeded. - Upload not ready (status is INITIALIZED), waiting for 5 seconds. - Scheduling a run. - Checking if the run succeeded. - Run not completed (status is SCHEDULING), waiting for 60 seconds. - Run not completed (status is RUNNING), waiting for 60 seconds. - Run not completed (status is RUNNING), waiting for 60 seconds. - Run not completed (status is RUNNING), waiting for 60 seconds. - Run not completed (status is RUNNING), waiting for 60 seconds. - Run not completed (status is RUNNING), waiting for 60 seconds. - Run completed: PASSED - ``` - -### Available commands - - You can use the `--help` command to get a list of all available options: - - ``` - $ python devicefarm.py --help - usage: Device Farm Runner [-h] [--project-arn PROJECT_ARN] - [--device-pool-arn DEVICE_POOL_ARN] - [--app-apk-path APP_APK_PATH] - [--test-apk-path TEST_APK_PATH] - - Runs the Espresso tests on AWS Device Farm. - - optional arguments: - -h, --help show this help message and exit - --project-arn PROJECT_ARN - The project ARN (Amazon Resource Name) (default: None) - --device-pool-arn DEVICE_POOL_ARN - The device pool ARN (Amazon Resource Name) (default: - None) - --app-apk-path APP_APK_PATH - Path to the app APK (default: None) - --test-apk-path TEST_APK_PATH - Path to the tests APK (default: None) - ``` +To run tests on AWS device farm you need to execute `./gradlew devicefarmUpload`. +You can configure the different steps in the testapp `build.gradle`. +AWS credentials are found in bitrise. diff --git a/platform/android/tests/scripts/devicefarm.py b/platform/android/tests/scripts/devicefarm.py deleted file mode 100644 index 6c0a337403..0000000000 --- a/platform/android/tests/scripts/devicefarm.py +++ /dev/null @@ -1,132 +0,0 @@ -''' -Uploads an APK and a test APK to AWS Device Farm and runs the Espresso tests. -Exit code is 0 if all tests pass, 1 otherwise. See README.md for details. -''' - -from time import sleep -import argparse -import boto3 -import requests - -''' -Parser -''' - -parser = argparse.ArgumentParser( - prog='Device Farm Runner', - description='Runs the Espresso tests on AWS Device Farm.', - formatter_class=argparse.ArgumentDefaultsHelpFormatter, - add_help=True) - -parser.add_argument('--project-arn', - type=str, help='The project ARN (Amazon Resource Name)') -parser.add_argument('--device-pool-arn', - type=str, help='The device pool ARN (Amazon Resource Name)') -parser.add_argument('--app-apk-path', - type=str, help='Path to the app APK') -parser.add_argument('--test-apk-path', - type=str, help='Path to the tests APK') - -args = vars(parser.parse_args()) -project_arn = args.get('project_arn') -device_pool_arn = args.get('device_pool_arn') -app_apk_path = args.get('app_apk_path') -test_apk_path = args.get('test_apk_path') - -# Check -if not project_arn or not device_pool_arn: - raise Exception('You need to set both the project and the device pool ARN.') -elif not app_apk_path or not test_apk_path: - raise Exception('You need to set both the app and test APK path.') - -''' -The AWS Device Farm client -''' - -client = boto3.client('devicefarm', region_name='us-west-2') - -''' -Methods -''' - -def upload_apk(apk_name, apk_path, apk_type): - print 'Starting upload: %s' % apk_type - result = client.create_upload( - projectArn=project_arn, name=apk_name, type=apk_type) - presigned_url = result.get('upload').get('url') - upload_arn = result.get('upload').get('arn') - - # PUT the file content and wait - put_file(apk_path=apk_path, presigned_url=presigned_url) - wait_for_upload(upload_arn=upload_arn) - return upload_arn - -def put_file(apk_path, presigned_url): - print 'Uploading: %s' % apk_path - with open(apk_path, 'rb') as f: - data = f.read() - result = requests.put(presigned_url, data=data) - if result.status_code != 200: - raise Exception( - 'PUT failed with status code: %s' % result.status_code) - -def wait_for_upload(upload_arn): - print 'Checking if the upload succeeded.' - succeeded = False - while not succeeded: - result = client.get_upload(arn=upload_arn) - status = result.get('upload').get('status') - succeeded = (status == 'SUCCEEDED') - if status == 'FAILED': - raise Exception('Upload failed.') - elif not succeeded: - print 'Upload is not ready (status is %s), waiting for 5 seconds.' % status - sleep(5) - -def schedule_run(app_arn, test_arn): - print 'Scheduling a run.' - result = client.schedule_run( - projectArn=project_arn, - appArn=app_arn, - devicePoolArn=device_pool_arn, - name='automated_run', - test={ 'type': 'INSTRUMENTATION', 'testPackageArn': test_arn}) - run_arn = result.get('run').get('arn') - return_code = wait_for_run(run_arn=run_arn) - return return_code - -def wait_for_run(run_arn): - print 'Checking if the run succeeded.' - return_code = '' - succeeded = False - while not succeeded: - result = client.get_run(arn=run_arn) - status = result.get('run').get('status') - return_code = result.get('run').get('result') - succeeded = (status == 'COMPLETED') - if not succeeded: - print 'Run not completed (status is %s), waiting for 60 seconds.' % status - sleep(60) - return return_code - -''' -Main flow -''' - -# 1. Upload the app APK -app_arn = upload_apk( - apk_name='app-debug-unaligned.apk', - apk_path=app_apk_path, - apk_type='ANDROID_APP') - -# 2. Upload the test APK -test_arn = upload_apk( - apk_name='app-debug-androidTest-unaligned.apk', - apk_path=test_apk_path, - apk_type='INSTRUMENTATION_TEST_PACKAGE') - -# 3. Schedule the run -return_code = schedule_run(app_arn=app_arn, test_arn=test_arn) -exit_code = 0 if return_code == 'PASSED' else 1 -print 'Run completed: %s' % return_code -exit(exit_code)
\ No newline at end of file |