The nova.utils.spawn and spawn_n methods transport
the context (and profiling information) to the
newly created threads. But the same isn't done
when submitting work to thread-pools in the
ComputeManager.

The code doing that for spawn and spawn_n
is extracted to a new function
and called to submit the work to the thread-pools.

Closes-Bug: #1962574
Change-Id: I9085deaa8cf0b167d87db68e4afc4a463c00569c

Currently nova.utils.tpool_execute() only works by chance. And as the
bug report shows there are env where it fails.

The nova.utils.tpool_execute() call tries to uses eventlet.tpool.execute
but the tpool module is not imported by the utils module only eventlet.
In devstack it works by chance as the wsgi init actually imports
eventlet.tpool indirectly via:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 21, in <module>
    from nova.api.openstack.compute.routes import APIRouterV21 # noqa
  File "/opt/stack/nova/nova/api/openstack/compute/routes.py", line 20, in <module>
    from nova.api.openstack.compute import admin_actions
  File "/opt/stack/nova/nova/api/openstack/compute/admin_actions.py", line 17, in <module>
    from nova.api.openstack import common
  File "/opt/stack/nova/nova/api/openstack/common.py", line 27, in <module>
    from nova.compute import task_states
  File "/opt/stack/nova/nova/compute/task_states.py", line 26, in <module>
    from nova.objects import fields
  File "/opt/stack/nova/nova/objects/fields.py", line 24, in <module>
    from nova.network import model as network_model
  File "/opt/stack/nova/nova/network/model.py", line 23, in <module>
    from nova import utils
  File "/opt/stack/nova/nova/utils.py", line 39, in <module>
    from oslo_concurrency import processutils
  File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 57, in <module>
    from eventlet import tpool

This was broken since I8dbc579e0037969aab4f2bb500fccfbde4190726. This
patch adds the correct import statement.

Change-Id: Ic46345ceeb445164aea6ae9b35c457c6150765f6
Closes-Bug: #1915400

The commit I168fffac8002f274a905cfd53ac4f6c9abe18803 added a wrapper
around fasteners.ReaderWriterLock to fix up an issue with eventlet. But
the wrapper was added to nova.utils module that is use not only by the
nova tests but also the nova production code. This made the fixture
library a dependency of the nova production code. While the current
ReaderWriterLock usage only limited to the nova test sub tree. The
I712f88fc1b6053fe6d1f13e708f3bd8874452a8f commit fix the issue of not
having fixtures in the nova requirements.txt. However I think a better
fix is to move the wrapper to the test subtree instead. This patch does
that and restores the state of the requirements.txt

Change-Id: I6903ce53b9b91325f7268cf2ebd02e4488579560
Related-Bug: #1958075

This is a follow up change to I168fffac8002f274a905cfd53ac4f6c9abe18803
which added a hackaround to enable our tests to pass with
fasteners>=0.15 which was upgraded recently as part of a
openstack/requirements update.

The ReaderWriterLock from fasteners (and thus lockutils) cannot work
correctly with eventlet patched code, so this adds a wrapper containing
the aforementioned hackaround along with a hacking check to do our best
to ensure that future use of ReaderWriterLock will be through the
wrapper.

Change-Id: Ia7bcb40a21a804c7bc6b74f501d95ce2a88b09b5

autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.

This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.

This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.

The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.

Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d

inspect.getargspec() is deprecated since py3

[1] https://docs.python.org/3/library/inspect.html#inspect.getargspec

Change-Id: I6fbdf942dd192351e135430997c15ea92e7fb8af

When creating an instance, nova sanitizes the 'instance.name' attribute,
using the 'nova.utils.sanitize_hostname' function, and stores this as
the 'instance.hostname' attribute. If neutron has the DNS extension
enabled then this attribute will be reported as the 'dns_name' value
when attaching ports [1].

Nova does not currently remove or replace periods in the name as part of
this sanitization. This results in the hostname being identified as a
(FQDN) fully qualified domain name, with all the constraints placed on
FQDNs [2]. This can be problematic for instances with common names such
as 'ubuntu18.04' or 'test.a', which aren't valid since TLDs must start
with an alpha character and must be greater than 1 character long,
respectively. Attempting to boot instances with such names can result in
failures like the below when DNS integration is enabled in neutron:

  RescheduledException: Build of instance foo was re-scheduled:
  Invalid input for dns_name.
  Reason: 'test-ubuntu-20.04' not a valid PQDN or FQDN.
  Reason: TLD '04' must not be all numeric.

Start replacing these periods with hyphens. This is a change in behavior
and may affect users who are relying on this inadvertent support for
instance names as FQDNs when using cloud-init, however, a quick poll on
openstack-discuss [3] suggests the people that are using FQDN-like
instance names are setting hostnames in the guest explicitly and not
relying on the information provided via the nova metadata service,
meaning this will not affect them.

[1] https://docs.openstack.org/neutron/victoria/admin/config-dns-int.html
[2] https://stackoverflow.com/a/53875771/613428
[3] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019113.html

Change-Id: I2fac7f919e9ddd6b0925d3e5d04e61b2ba1b8c82
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Closes-Bug: #1581977

We have discovered that if an exception is raised at any point during
the running of the init_application WSGI script in an apache/mod_wsgi
Daemon Mode environment, it will prompt apache/mod_wsgi to re-run the
script without starting a fresh python process. Because we initialize
global data structures during app init, subsequent runs of the script
blow up as some global data do *not* support re-initialization. It is
anyway not safe to assume that init of global data is safe to run
multiple times.

This mod_wsgi behavior appears to be a special situation that does not
behave the same as a normal reload in Daemon Mode as the script file is
being reloaded upon failure instead of the daemon process being
shutdown and restarted as described in the documentation [1].

In order to handle this situation, we can move the initialization of
global data structures to a helper method that is decorated to run only
once per python interpreter instance. This way, we will not attempt to
re-initialize global data that are not safe to init more than once.

Co-Authored-By: Michele Baldessari <michele@redhat.com>
Co-Authored-By: melanie witt <melwittt@gmail.com>

Closes-Bug: #1882094

[1] https://modwsgi.readthedocs.io/en/develop/user-guides/reloading-source-code.html#reloading-in-daemon-mode

Change-Id: I2bd360dcc6501feea7baf02d4510b282205fc061

Replace six.text_type with str.
A subsequent patch will replace other six.text_type.

Change-Id: I23bb9e539d08f5c6202909054c2dd49b6c7a7a0e
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>