Use the new API that is consistent with
the existing API instead of instantiating the client
class directly.

This was introduced in release 14.1.0 here [1] and
added into oslo.messaging here [2]

[1] https://review.opendev.org/c/openstack/requirements/+/869340
[2] https://review.opendev.org/c/openstack/oslo.messaging/+/862419

Change-Id: If1128fe0faacef757ed10023c00a67e5ec0440bd

Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib. Most of this
is autogenerated, as described below, but there is one manual change
necessary:

nova/tests/functional/regressions/test_bug_1781286.py
  We need to avoid using 'fixtures.MockPatch' since fixtures is using
  'mock' (the library) under the hood and a call to 'mock.patch.stop'
  found in that test will now "stop" mocks from the wrong library. We
  have discussed making this configurable but the option proposed isn't
  that pretty [1] so this is better.

The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:

  import glob

  for path in glob.glob('nova/tests/**/*.py', recursive=True):
      with open(path) as fh:
          lines = fh.readlines()
      if 'import mock\n' not in lines:
          continue
      import_group_found = False
      create_first_party_group = False
      for num, line in enumerate(lines):
          line = line.strip()
          if line.startswith('import ') or line.startswith('from '):
              tokens = line.split()
              for lib in (
                  'ddt', 'six', 'webob', 'fixtures', 'testtools'
                  'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
              ):
                  if lib in tokens[1]:
                      create_first_party_group = True
                      break
              if create_first_party_group:
                  break
              import_group_found = True
          if not import_group_found:
              continue
          if line.startswith('import ') or line.startswith('from '):
              tokens = line.split()
              if tokens[1] > 'unittest':
                  break
              elif tokens[1] == 'unittest' and (
                  len(tokens) == 2 or tokens[4] > 'mock'
              ):
                  break
          elif not line:
              break
      if create_first_party_group:
          lines.insert(num, 'from unittest import mock\n\n')
      else:
          lines.insert(num, 'from unittest import mock\n')
      del lines[lines.index('import mock\n')]
      with open(path, 'w+') as fh:
          fh.writelines(lines)

Note that we cannot remove mock from our requirements files yet due to
importing pypowervm unit test code in nova unit tests. This library
still uses the mock lib, and since we are importing test code and that
lib (correctly) only declares mock in its test-requirements.txt, mock
would not otherwise be installed and would cause errors while loading
nova unit test code.

[1] https://github.com/testing-cabal/fixtures/pull/49

Change-Id: Id5b04cf2f6ca24af8e366d23f15cf0e5cac8e1cc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

The 'nova.exception_wrapper.wrap_exception' decorator accepted either a
pre-configured notifier or a 'get_notifier' function, but the forget was
never provided and the latter was consistently a notifier created via a
call to 'nova.rpc.get_notifier'. Simplify things by passing the
arguments relied by 'get_notifier' into 'wrap_exception', allowing the
latter to create the former for us.

While doing this rework, it became obvious that 'get_notifier' accepted
a 'published_id' that is never provided nowadays, so that is dropped. In
addition, a number of calls to 'get_notifier' were passing in
'host=CONF.host', which duplicated the default value for this parameter
and is therefore unnecessary. Finally, the unit tests are split up by
file, as they should be.

Change-Id: I89e1c13e8a0df18594593b1e80c60d177e0d9c4c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

Replace six.text_type with str.
This patch completes six removal.

Change-Id: I779bd1446dc1f070fa5100ccccda7881fa508d79
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>

We're occasionally seeing stacktraces like this in our tests:

  Fatal Python error: Cannot recover from stack overflow.

  Current thread 0x00007fe66549f740 (most recent call first):
    File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614 in _get
    File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2183 in __getattr__
    File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614 in _get
    File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2183 in __getattr__
    File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614 in _get
    ...

From a look at the oslo.config source, this seems to be occurring
because 'ConfigOpts.__cache' is somehow undefined, which results in the
'_get' method attempting to call '__getattr__' [1], which calls '_get'
[2], which calls '__getattr__' and so on.

The exact reason this is happening isn't clear, but what is clear is
that how we handle global config options in the tests that are failing
is very strange and potentially subject to race conditions. We have a
clear pattern for mocking everything and anything - the mock module -
and we should be using this here. Start doing so, reworking a lot of the
tests in the process, in order to avoid messing with oslo.config and
triggering the issue entirely.

[1] https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2614
[2] https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2183

Change-Id: I468cef94185a1b59f379ca527050450e03664c67
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Closes-Bug: #1825435

The CheatingSerializer fixture used in nova tests keeps
the RequestContext.db_connection set on the context object
which otherwise wouldn't normally happen. The context object
can show up in error notification payloads because of how
nova.exception_wrapper.wrap_exception works. That payload
is eventually serialized for notifications and since the
cheated RequestContext.db_connection is set and cannot be
serialized, it results in a UserWarning from the
jsonutils.to_primitive method (called via JsonPayloadSerializer).

This will eventually result in failures when that UserWarning
is made into an error.

To fix this, we can pass a fallback method to to_primitive()
which will serialize a RequestContext object the same way that
RequestContextSerializer serializes a context - by simply
converting it to dict form.

Since this only affects test runs, because of using the
CheatingSerializer fixture, it should have no impact on
runtime serializations.

Error logging is added to the FakeNotifier since it's hard
to know what is wrong in the payload unless it is logged.

Also, the WarningsFixture is updated to make sure we don't
introduce new UserWarnings for the serialization issue.

The jsonutils.to_primitive() fallback method was added to
oslo.serialization via commit cdb2f60d26e3b65b6370f87b2e9864045651c117
in 2.21.1 so we have to bump our minimum required version
of that library as well.

Change-Id: Id9f960a0c7c8897dbb9edf53b4723154341412d6
Closes-Bug: #1799249

This change makes nova configure oslo.messaging's active call monitoring
feature if the operator increases the rpc_response_timeout configuration
option beyond the default of 60 seconds. If this happens, oslo.messaging will
heartbeat actively-running calls to indicate that they are still running,
avoiding a false timeout at the shorter interval, while still detecting
actual dead-service failures before the longer timeout value.

In addition, this adds a long_rpc_timeout configuration option that we
can use for known-to-run-long operations separately from the base
rpc_response_timeout value, and pre_live_migration() is changed to use
this, as it is known to suffer from early false timeouts.

Depends-On: Iecb7bef61b3b8145126ead1f74dbaadd7d97b407
Change-Id: Icb0bdc6d4ce4524341e70e737eafcb25f346d197

By not inheriting from the base nova test case, we
lose things like timeouts.

This makes the TestRPC test class inherit from the
base test case and still avoid using the RPCFixture.

Change-Id: Id65e15a57175bbdd9c84851b1b6716e6a1f2cfb8
Related-Bug: #1685333

The get_transport method was deprecated in the oslo.messaging
5.24.2 release in favor of get_rpc_transport and
get_notification_transport. We are already using get_notification_transport,
we just need to change to use get_rpc_transport.

Change-Id: Ie276fedef35f411cafbaf25c2148fea42ae01410

This removes the two deprecated by_instance() and by_host() methods
in ClientRouter and replaces them with calls to a single client()
routine. We stopped using the instance and host parameters in these
methods in an earlier patch to remove the caching in the router.

Related to blueprint cells-aware-api

Change-Id: Ifa7cd99a442ef0f8e9ad4a6b52982d0fedf1cd7e