diff options
| -rw-r--r-- | HACKING.rst | 48 | ||||
| -rw-r--r-- | README.rst | 54 | ||||
| -rw-r--r-- | REVIEWING.rst | 24 | ||||
| -rw-r--r-- | doc/source/configuration.rst | 62 | ||||
| -rw-r--r-- | doc/source/write_tests.rst | 18 | ||||
| -rw-r--r-- | tempest/README.rst | 15 | ||||
| -rw-r--r-- | tempest/api/README.rst | 5 | ||||
| -rw-r--r-- | tempest/scenario/README.rst | 6 | ||||
| -rw-r--r-- | tempest/tests/README.rst | 12 |
9 files changed, 137 insertions, 107 deletions
diff --git a/HACKING.rst b/HACKING.rst index 77a8612ef..e5f45ac1b 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -102,20 +102,20 @@ to work even if just one ``test_method`` is selected for execution. Service Tagging --------------- Service tagging is used to specify which services are exercised by a particular -test method. You specify the services with the tempest.test.services decorator. -For example: +test method. You specify the services with the ``tempest.test.services`` +decorator. For example: @services('compute', 'image') Valid service tag names are the same as the list of directories in tempest.api that have tests. -For scenario tests having a service tag is required. For the api tests service -tags are only needed if the test method makes an api call (either directly or +For scenario tests having a service tag is required. For the API tests service +tags are only needed if the test method makes an API call (either directly or indirectly through another service) that differs from the parent directory -name. For example, any test that make an api call to a service other than nova -in tempest.api.compute would require a service tag for those services, however -they do not need to be tagged as compute. +name. For example, any test that make an API call to a service other than Nova +in ``tempest.api.compute`` would require a service tag for those services, +however they do not need to be tagged as ``compute``. Test fixtures and resources --------------------------- @@ -198,7 +198,7 @@ will expand the guideline based on our discussion and experience. Test skips because of Known Bugs -------------------------------- If a test is broken because of a bug it is appropriate to skip the test until -bug has been fixed. You should use the skip_because decorator so that +bug has been fixed. You should use the ``skip_because`` decorator so that Tempest's skip tracking tool can watch the bug status. Example:: @@ -229,7 +229,7 @@ parallel. require admin privileges are outside of projects. - Races between methods in the same class are not a problem because - parallelization in tempest is at the test class level, but if there is a json + parallelization in Tempest is at the test class level, but if there is a json and xml version of the same test class there could still be a race between methods. @@ -238,8 +238,8 @@ parallel. avoided to prevent resource conflicts. - If the execution of a set of tests is required to be serialized then locking - can be used to perform this. See AggregatesAdminTest in - tempest.api.compute.admin for an example of using locking. + can be used to perform this. See usage of ``LockFixture`` for examples of + using locking. Sample Configuration File ------------------------- @@ -251,7 +251,7 @@ regenerated. This can be done running:: Unit Tests ---------- -Unit tests are a separate class of tests in tempest. They verify tempest +Unit tests are a separate class of tests in Tempest. They verify Tempest itself, and thus have a different set of guidelines around them: 1. They can not require anything running externally. All you should need to @@ -321,8 +321,8 @@ format of the metadata looks like:: Tempest.lib includes a ``check-uuid`` tool that will test for the existence and uniqueness of idempotent_id metadata for every test. If you have -tempest installed you run the tool against Tempest by calling from the -tempest repo:: +Tempest installed you run the tool against Tempest by calling from the +Tempest repo:: check-uuid @@ -337,7 +337,7 @@ one:: check-uuid --fix -The ``check-uuid`` tool is used as part of the tempest gate job +The ``check-uuid`` tool is used as part of the Tempest gate job to ensure that all tests have an ``idempotent_id`` decorator. Branchless Tempest Considerations @@ -350,7 +350,7 @@ branches are also gated by the Tempest master branch, which also means that proposed commits to Tempest must work against both the master and all the currently supported stable branches of the projects. As such there are a few special considerations that have to be accounted for when pushing new changes -to tempest. +to Tempest. 1. New Tests for new features ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -370,14 +370,22 @@ be able to merge. When trying to land a bug fix which changes a tested API you'll have to use the following procedure:: - - Propose change to the project, get a +2 on the change even with failing - - Propose skip on Tempest which will only be approved after the + 1. Propose change to the project, get a +2 on the change even with failing + 2. Propose skip on Tempest which will only be approved after the corresponding change in the project has a +2 on change - - Land project change in master and all open stable branches (if required) - - Land changed test in Tempest + 3. Land project change in master and all open stable branches (if required) + 4. Land changed test in Tempest Otherwise the bug fix won't be able to land in the project. +Handily, `Zuul’s cross-repository dependencies +<https://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies>`_. +can be leveraged to do without step 2 and to have steps 3 and 4 happen +"atomically". To do that, make the patch written in step 1 to depend (refer to +Zuul's documentation above) on the patch written in step 4. The commit message +for the Tempest change should have a link to the Gerrit review that justifies +that change. + 3. New Tests for existing features ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/README.rst b/README.rst index a7eab2f99..2e13fec5a 100644 --- a/README.rst +++ b/README.rst @@ -23,7 +23,7 @@ Design Principles Tempest Design Principles that we strive to live by. - Tempest should be able to run against any OpenStack cloud, be it a - one node devstack install, a 20 node lxc cloud, or a 1000 node kvm + one node DevStack install, a 20 node LXC cloud, or a 1000 node KVM cloud. - Tempest should be explicit in testing features. It is easy to auto discover features of a cloud incorrectly, and give people an @@ -65,13 +65,13 @@ as it is simpler, and quicker to work with. $ pip install tempest/ This can be done within a venv, but the assumption for this guide is that - the Tempest cli entry point will be in your shell's PATH. + the Tempest CLI entry point will be in your shell's PATH. -#. Installing Tempest may create a /etc/tempest dir, however if one isn't - created you can create one or use ~/.tempest/etc or ~/.config/tempest in - place of /etc/tempest. If none of these dirs are created tempest will create - ~/.tempest/etc when it's needed. The contents of this dir will always - automatically be copied to all etc/ dirs in local workspaces as an initial +#. Installing Tempest may create a ``/etc/tempest dir``, however if one isn't + created you can create one or use ``~/.tempest/etc`` or ``~/.config/tempest`` in + place of ``/etc/tempest``. If none of these dirs are created Tempest will create + ``~/.tempest/etc`` when it's needed. The contents of this dir will always + automatically be copied to all ``etc/`` dirs in local workspaces as an initial setup step. So if there is any common configuration you'd like to be shared between local Tempest workspaces it's recommended that you pre-populate it before running ``tempest init``. @@ -90,12 +90,12 @@ as it is simpler, and quicker to work with. is that you'll create a new working directory for each to maintain separate configuration files and local artifact storage for each. -#. Then cd into the newly created working dir and also modify the local - config files located in the etc/ subdir created by the ``tempest init`` - command. Tempest is expecting a tempest.conf file in etc/ so if only a +#. Then ``cd`` into the newly created working dir and also modify the local + config files located in the ``etc/`` subdir created by the ``tempest init`` + command. Tempest is expecting a ``tempest.conf`` file in etc/ so if only a sample exists you must rename or copy it to tempest.conf before making any changes to it otherwise Tempest will not know how to load it. For - details on configuring tempest refer to the :ref:`tempest-configuration`. + details on configuring Tempest refer to the :ref:`tempest-configuration`. #. Once the configuration is done you're now ready to run Tempest. This can be done using the :ref:`tempest_run` command. This can be done by either @@ -124,8 +124,8 @@ Library Tempest exposes a library interface. This interface is a stable interface and should be backwards compatible (including backwards compatibility with the old tempest-lib package, with the exception of the import). If you plan to -directly consume tempest in your project you should only import code from the -tempest library interface, other pieces of tempest do not have the same +directly consume Tempest in your project you should only import code from the +Tempest library interface, other pieces of Tempest do not have the same stable interface and there are no guarantees on the Python API unless otherwise stated. @@ -137,7 +137,7 @@ Release Versioning shows what changes have been released on each version. Tempest's released versions are broken into 2 sets of information. Depending on -how you intend to consume tempest you might need +how you intend to consume Tempest you might need The version is a set of 3 numbers: @@ -146,12 +146,12 @@ X.Y.Z While this is almost `semver`_ like, the way versioning is handled is slightly different: -X is used to represent the supported OpenStack releases for tempest tests -in-tree, and to signify major feature changes to tempest. It's a monotonically +X is used to represent the supported OpenStack releases for Tempest tests +in-tree, and to signify major feature changes to Tempest. It's a monotonically increasing integer where each version either indicates a new supported OpenStack release, the drop of support for an OpenStack release (which will coincide with the upstream stable branch going EOL), or a major feature lands (or is removed) -from tempest. +from Tempest. Y.Z is used to represent library interface changes. This is treated the same way as minor and patch versions from `semver`_ but only for the library @@ -166,16 +166,16 @@ Configuration Detailed configuration of Tempest is beyond the scope of this document see :ref:`tempest-configuration` for more details on configuring -Tempest. The etc/tempest.conf.sample attempts to be a self-documenting version -of the configuration. +Tempest. The ``etc/tempest.conf.sample`` attempts to be a self-documenting +version of the configuration. You can generate a new sample tempest.conf file, run the following command from the top level of the Tempest directory:: $ tox -e genconfig -The most important pieces that are needed are the user ids, openstack -endpoint, and basic flavors and images needed to run tests. +The most important pieces that are needed are the user ids, OpenStack +endpoints, and basic flavors and images needed to run tests. Unit Tests ---------- @@ -190,13 +190,13 @@ should only be run on the unit test directory. The default value of OS_TEST_PATH is OS_TEST_PATH=./tempest/test_discover which will only run test discover on the Tempest suite. -Alternatively, there are the py27 and py34 tox jobs which will run the unit +Alternatively, there are the py27 and py35 tox jobs which will run the unit tests with the corresponding version of python. Python 2.6 ---------- -Starting in the kilo release the OpenStack services dropped all support for +Starting in the Kilo release the OpenStack services dropped all support for python 2.6. This change has been mirrored in Tempest, starting after the tempest-2 tag. This means that proposed changes to Tempest which only fix python 2.6 compatibility will be rejected, and moving forward more features not @@ -208,8 +208,8 @@ that has python 2.7) Python 3.x ---------- -Starting during the Pike cycle Tempest has a gating CI job that runs tempest -with Python 3. Any tempest release after 15.0.0 should fully support running +Starting during the Pike cycle Tempest has a gating CI job that runs Tempest +with Python 3. Any Tempest release after 15.0.0 should fully support running under Python 3 as well as Python 2.7. Legacy run method @@ -239,10 +239,10 @@ and reference data to be used in testing. .. note:: - If you have a running devstack environment, Tempest will be + If you have a running DevStack environment, Tempest will be automatically configured and placed in ``/opt/stack/tempest``. It will have a configuration file already set up to work with your - devstack installation. + DevStack installation. Tempest is not tied to any single test runner, but `testr`_ is the most commonly used tool. Also, the nosetests test runner is **not** recommended to run Tempest. diff --git a/REVIEWING.rst b/REVIEWING.rst index 5807d50f7..7d28320b4 100644 --- a/REVIEWING.rst +++ b/REVIEWING.rst @@ -21,6 +21,15 @@ manual/developers.html#cross-repository-dependencies>`_ or a simple link to a Gerrit review. +Execution time +-------------- +While checking in the job logs that a new test is actually executed, also +pay attention to the execution time of that test. Keep in mind that each test +is going to be executed hundreds of time each day, because Tempest tests +run in many OpenStack projects. It's worth considering how important/critical +the feature under test is with how costly the new test is. + + Unit Tests ---------- @@ -48,6 +57,17 @@ the maintenance burden. Such changes should not be approved if it is easy to abstract the duplicated code into a function or method. +Tests overlap +------------- +When a new test is being proposed, question whether this feature is not already +tested with Tempest. Tempest has more than 1200 tests, spread amongst many +directories, so it's easy to introduce test duplication. For example, testing +volume attachment to a server could be a compute test or a volume test, depending +on how you see it. So one must look carefully in the entire code base for possible +overlap. As a rule of thumb, the older a feature is, the more likely it's +already tested. + + Being explicit -------------- When tests are being added that depend on a configurable feature or extension, @@ -60,8 +80,8 @@ whether to skip or not. Configuration Options --------------------- -With the introduction of the tempest external test plugin interface we needed -to provide a stable contract for tempest's configuration options. This means +With the introduction of the Tempest external test plugin interface we needed +to provide a stable contract for Tempest's configuration options. This means we can no longer simply remove a configuration option when it's no longer used. Patches proposed that remove options without a deprecation cycle should not be approved. Similarly when changing default values with configuration we need diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst index 4accd946c..8f2865ad5 100644 --- a/doc/source/configuration.rst +++ b/doc/source/configuration.rst @@ -28,7 +28,7 @@ can be used to: - Run tests for admin APIs - Generate test credentials on the fly (see `Dynamic Credentials`_) -When keystone uses a policy that requires domain scoped tokens for admin +When Keystone uses a policy that requires domain scoped tokens for admin actions, the flag ``admin_domain_scope`` must be set to ``True``. The admin user configured, if any, must have a role assigned to the domain to be usable. @@ -39,7 +39,7 @@ This can be done using the accounts.yaml file (see number of users available to run tests with. You can specify the location of the file in the ``auth`` section in the tempest.conf file. To see the specific format used in the file please refer to -the accounts.yaml.sample file included in Tempest. +the ``accounts.yaml.sample`` file included in Tempest. Keystone Connection Info ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -47,18 +47,17 @@ In order for Tempest to be able to talk to your OpenStack deployment you need to provide it with information about how it communicates with keystone. This involves configuring the following options in the ``identity`` section: - #. ``auth_version`` - #. ``uri`` - #. ``uri_v3`` + - ``auth_version`` + - ``uri`` + - ``uri_v3`` The ``auth_version`` option is used to tell Tempest whether it should be using -keystone's v2 or v3 api for communicating with keystone. The two uri options are +Keystone's v2 or v3 api for communicating with Keystone. The two uri options are used to tell Tempest the url of the keystone endpoint. The ``uri`` option is -used for keystone v2 request and ``uri_v3`` is used for keystone v3. You want to +used for Keystone v2 request and ``uri_v3`` is used for Keystone v3. You want to ensure that which ever version you set for ``auth_version`` has its uri option defined. - Credential Provider Mechanisms ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -95,7 +94,7 @@ configured to use dynamic credentials. When the ``admin_domain_scope`` option is set to ``True``, provisioned admin accounts will be assigned a role on domain configured in ``default_credentials_domain_name``. This will make the accounts provisioned -usable in a cloud where domain scoped tokens are required by keystone for +usable in a cloud where domain scoped tokens are required by Keystone for admin operations. Note that the initial pre-provision admin accounts, configured in tempest.conf, must have a role on the same domain as well, for Dynamic Credentials to work. @@ -140,10 +139,10 @@ used do not have any pre-existing resources created. Tempest assumes all tenants it's using are empty and may sporadically fail if there are unexpected resources present. -When the keystone in the target cloud requires domain scoped tokens to +When the Keystone in the target cloud requires domain scoped tokens to perform admin actions, all pre-provisioned admin users must have a role assigned on the domain where test accounts a provisioned. -The option ``admin_domain_scope`` is used to tell tempest that domain scoped +The option ``admin_domain_scope`` is used to tell Tempest that domain scoped tokens shall be used. ``default_credentials_domain_name`` is the domain where test accounts are expected to be provisioned if no domain is specified. @@ -167,7 +166,7 @@ for doing this: #. ``flavor_ref_alt`` Both of these options are in the ``compute`` section of the config file and take -in the flavor id (not the name) from nova. The ``flavor_ref`` option is what +in the flavor id (not the name) from Nova. The ``flavor_ref`` option is what will be used for booting almost all of the guests; ``flavor_ref_alt`` is only used in tests where two different-sized servers are required (for example, a resize test). @@ -185,7 +184,7 @@ servers. There are two options in the compute section just like with flavors: #. ``image_ref`` #. ``image_ref_alt`` -Both options are expecting an image id (not name) from nova. The ``image_ref`` +Both options are expecting an image id (not name) from Nova. The ``image_ref`` option is what will be used for booting the majority of servers in Tempest. ``image_ref_alt`` is used for tests that require two images such as rebuild. If two images are not available you can set both options to the same image id and @@ -224,7 +223,7 @@ small footprint. Networking ---------- OpenStack has a myriad of different networking configurations possible and -depending on which of the two network backends, nova-network or neutron, you are +depending on which of the two network backends, nova-network or Neutron, you are using things can vary drastically. Due to this complexity Tempest has to provide a certain level of flexibility in its configuration to ensure it will work against any cloud. This ends up causing a large number of permutations in @@ -302,21 +301,21 @@ cases it shouldn't matter because nova-network should have no problem booting a server with multiple networks. If this is not the case for your cloud then using an accounts file is recommended because it provides the necessary flexibility to describe your configuration. Dynamic credentials is not able to dynamically -allocate things as necessary if neutron is not enabled. +allocate things as necessary if Neutron is not enabled. -With neutron and dynamic credentials enabled there should not be any additional +With Neutron and dynamic credentials enabled there should not be any additional configuration necessary to enable Tempest to create servers with working networking, assuming you have properly configured the ``network`` section to -work for your cloud. Tempest will dynamically create the neutron resources +work for your cloud. Tempest will dynamically create the Neutron resources necessary to enable using servers with that network. Also, just as with the -accounts file, if you specify a fixed network name while using neutron and +accounts file, if you specify a fixed network name while using Neutron and dynamic credentials it will enable running tests which require a static network and it will additionally be used as a fallback for server creation. However, unlike accounts.yaml this should never be triggered. However, there is an option ``create_isolated_networks`` to disable dynamic credentials's automatic provisioning of network resources. If this option is set -to False you will have to either rely on there only being a single/default +to ``False`` you will have to either rely on there only being a single/default network available for the server creation, or use ``fixed_network_name`` to inform Tempest which network to use. @@ -336,16 +335,16 @@ To enable remote access to servers, there are 3 options at a minimum that are us The ``run_validation`` is used to enable or disable ssh connectivity for all tests (with the exception of scenario tests which do not have a flag for -enabling or disabling ssh) To enable ssh connectivity this needs be set to ``true``. +enabling or disabling ssh) To enable ssh connectivity this needs be set to ``True``. -The ``connect_method`` option is used to tell tempest what kind of IP to use for +The ``connect_method`` option is used to tell Tempest what kind of IP to use for establishing a connection to the server. Two methods are available: ``fixed`` and ``floating``, the later being set by default. If this is set to floating -tempest will create a floating ip for the server before attempted to connect +Tempest will create a floating ip for the server before attempted to connect to it. The IP for the floating ip is what is used for the connection. For the ``auth_method`` option there is currently, only one valid option, -``keypair``. With this set to ``keypair`` tempest will create an ssh keypair +``keypair``. With this set to ``keypair`` Tempest will create an ssh keypair and use that for authenticating against the created server. Configuring Available Services @@ -359,8 +358,8 @@ on the available services. The ``service_available`` section of the config file is used to set which services are available. It contains a boolean option for each service (except -for keystone which is a hard requirement) set it to True if the service is -available or False if it is not. +for Keystone which is a hard requirement) set it to ``True`` if the service is +available or ``False`` if it is not. Service Catalog ^^^^^^^^^^^^^^^ @@ -382,17 +381,18 @@ has multiple regions available and you need to specify a particular one to use a service you can set the ``region`` option in that service's section. It should also be noted that the default values for these options are set -to what devstack uses (which is a de facto standard for service catalog +to what DevStack uses (which is a de facto standard for service catalog entries). So often nothing actually needs to be set on these options to enable communication to a particular service. It is only if you are either not using -the same ``catalog_type`` as devstack or you want Tempest to talk to a different -endpoint type instead of publicURL for a service that these need to be changed. +the same ``catalog_type`` as DevStack or you want Tempest to talk to a different +endpoint type instead of ``publicURL`` for a service that these need to be +changed. .. note:: Tempest does not serve all kinds of fancy URLs in the service catalog. The service catalog should be in a standard format (which is going to be - standardized at the keystone level). + standardized at the Keystone level). Tempest expects URLs in the Service catalog in the following format: * ``http://example.com:1234/<version-info>`` @@ -420,7 +420,7 @@ for a feature (assuming one exists). Instead Tempest has to be explicitly configured as to which optional features are enabled. This is in order to prevent bugs in the discovery mechanisms from masking failures. -The service feature-enabled config sections are how Tempest addresses the +The service ``feature-enabled`` config sections are how Tempest addresses the optional feature question. Each service that has tests for optional features contains one of these sections. The only options in it are boolean options with the name of a feature which is used. If it is set to false any test which @@ -431,7 +431,7 @@ options refer to the sample config file. API Extensions ^^^^^^^^^^^^^^ The service feature-enabled sections often contain an ``api-extensions`` option -(or in the case of swift a ``discoverable_apis`` option). This is used to tell +(or in the case of Swift a ``discoverable_apis`` option). This is used to tell Tempest which api extensions (or configurable middleware) is used in your deployment. It has two valid config states: either it contains a single value ``all`` (which is the default) which means that every api extension is assumed diff --git a/doc/source/write_tests.rst b/doc/source/write_tests.rst index 4e3bfa2ec..aec55e94c 100644 --- a/doc/source/write_tests.rst +++ b/doc/source/write_tests.rst @@ -6,13 +6,13 @@ Tempest Test Writing Guide This guide serves as a starting point for developers working on writing new Tempest tests. At a high level tests in Tempest are just tests that conform to the standard python `unit test`_ framework. But there are several aspects of -that are unique to tempest and it's role as an integration test suite running +that are unique to Tempest and its role as an integration test suite running against a real cloud. .. _unit test: https://docs.python.org/3.6/library/unittest.html -.. note:: This guide is for writing tests in the tempest repository. While many - parts of this guide are also applicable to tempest plugins, not all +.. note:: This guide is for writing tests in the Tempest repository. While many + parts of this guide are also applicable to Tempest plugins, not all the APIs mentioned are considered stable or recommended for use in plugins. Please refer to :ref:`tempest_plugin` for details about writing plugins @@ -24,8 +24,8 @@ Adding a New TestCase The base unit of testing in Tempest is the `TestCase`_ (also called the test class). Each TestCase contains test methods which are the individual tests that will be executed by the test runner. But, the TestCase is the smallest self -contained unit for tests from the tempest perspective. It's also the level at -which tempest is parallel safe. In other words, multiple TestCases can be +contained unit for tests from the Tempest perspective. It's also the level at +which Tempest is parallel safe. In other words, multiple TestCases can be executed in parallel, but individual test methods in the same TestCase can not. Also, all test methods within a TestCase are assumed to be executed serially. As such you can use the test case to store variables that are shared between @@ -151,7 +151,7 @@ for tests cases that need a second user/project. You can also specify credentials with specific roles assigned. This is useful for cases where there are specific RBAC requirements hard coded into an API. The canonical example of this are swift tests which often want to test swift's -concepts of operator and reseller_admin. An actual example from tempest on how +concepts of operator and reseller_admin. An actual example from Tempest on how to do this is:: class PublicObjectTest(base.BaseObjectTest): @@ -187,7 +187,7 @@ requested credentials you can reference: | [$label, $role] | cls.os_roles_$label | +-------------------+---------------------+ -By default cls.os_primary is available since it is allocated in the base tempest test +By default cls.os_primary is available since it is allocated in the base Tempest test class (located in tempest/test.py). If your TestCase inherits from a different direct parent class (it'll still inherit from the BaseTestCase, just not directly) be sure to check if that class overrides allocated credentials. @@ -195,8 +195,8 @@ directly) be sure to check if that class overrides allocated credentials. Dealing with Network Allocation ''''''''''''''''''''''''''''''' -When neutron is enabled and a testing requires networking this isn't normally -automatically setup when a tenant is created. Since tempest needs isolated +When Neutron is enabled and a testing requires networking this isn't normally +automatically setup when a tenant is created. Since Tempest needs isolated tenants to function properly it also needs to handle network allocation. By default the base test class will allocate a network, subnet, and router automatically (this depends on the configured credential provider, for more diff --git a/tempest/README.rst b/tempest/README.rst index 0feec41e4..663653e4e 100644 --- a/tempest/README.rst +++ b/tempest/README.rst @@ -9,12 +9,13 @@ implementations for both correctness, as well as a burn in tool for OpenStack clouds. As such Tempest tests come in many flavors, each with their own rules -and guidelines. Below is the proposed Havana restructuring for Tempest +and guidelines. Below is the overview of the Tempest respository structure to make this clear. | tempest/ | api/ - API tests | scenario/ - complex scenario tests +| tests/ - unit tests for Tempest internals Each of these directories contains different types of tests. What belongs in each directory, the rules and examples for good tests, are @@ -24,8 +25,8 @@ documented in a README.rst file in the directory. ---------------------- API tests are validation tests for the OpenStack API. They should not -use the existing python clients for OpenStack, but should instead use -the tempest implementations of clients. Having raw clients let us +use the existing Python clients for OpenStack, but should instead use +the Tempest implementations of clients. Having raw clients let us pass invalid JSON to the APIs and see the results, something we could not get with the native clients. @@ -41,14 +42,14 @@ Scenario tests are complex "through path" tests for OpenStack functionality. They are typically a series of steps where complicated state requiring multiple services is set up exercised, and torn down. -Scenario tests should not use the existing python clients for OpenStack, -but should instead use the tempest implementations of clients. +Scenario tests should not use the existing Python clients for OpenStack, +but should instead use the Tempest implementations of clients. :ref:`unit_tests_field_guide` ----------------------------- Unit tests are the self checks for Tempest. They provide functional -verification and regression checking for the internal components of tempest. -They should be used to just verify that the individual pieces of tempest are +verification and regression checking for the internal components of Tempest. +They should be used to just verify that the individual pieces of Tempest are working as expected. diff --git a/tempest/api/README.rst b/tempest/api/README.rst index 91e6ad6b9..a79692219 100644 --- a/tempest/api/README.rst +++ b/tempest/api/README.rst @@ -13,7 +13,8 @@ portion of Tempest code is devoted to test cases that do exactly this. It's also important to test not only the expected positive path on APIs, but also to provide them with invalid data to ensure they fail -in expected and documented ways. Over the course of the OpenStack +in expected and documented ways. The latter type of tests is called +``negative tests`` in Tempest source code. Over the course of the OpenStack project Tempest has discovered many fundamental bugs by doing just this. @@ -22,7 +23,7 @@ enough data in the system. This means these tests might start by spinning up a server, image, etc, then operating on it. -Why are these tests in tempest? +Why are these tests in Tempest? ------------------------------- This is one of the core missions for the Tempest project, and where it diff --git a/tempest/scenario/README.rst b/tempest/scenario/README.rst index 38e0de967..c1dcccc0b 100644 --- a/tempest/scenario/README.rst +++ b/tempest/scenario/README.rst @@ -21,9 +21,9 @@ Any scenario test should have a real-life use case. An example would be: 4. create a snapshot of the vm -Why are these tests in tempest? +Why are these tests in Tempest? ------------------------------- -This is one of tempests core purposes, testing the integration between +This is one of Tempest's core purposes, testing the integration between projects. @@ -43,7 +43,7 @@ While we are looking for interaction of 2 or more services, be specific in your interactions. A giant "this is my data center" smoke test is hard to debug when it goes wrong. -A flow of interactions between glance and nova, like in the +A flow of interactions between Glance and Nova, like in the introduction, is a good example. Especially if it involves a repeated interaction when a resource is setup, modified, detached, and then reused later again. diff --git a/tempest/tests/README.rst b/tempest/tests/README.rst index e54d4c035..0587e7b88 100644 --- a/tempest/tests/README.rst +++ b/tempest/tests/README.rst @@ -7,16 +7,16 @@ What are these tests? --------------------- Unit tests are the self checks for Tempest. They provide functional -verification and regression checking for the internal components of tempest. -They should be used to just verify that the individual pieces of tempest are +verification and regression checking for the internal components of Tempest. +They should be used to just verify that the individual pieces of Tempest are working as expected. They should not require an external service to be running -and should be able to run solely from the tempest tree. +and should be able to run solely from the Tempest tree. -Why are these tests in tempest? +Why are these tests in Tempest? ------------------------------- These tests exist to make sure that the mechanisms that we use inside of -tempest to are valid and remain functional. They are only here for self -validation of tempest. +Tempest are valid and remain functional. They are only here for self +validation of Tempest. Scope of these tests |