| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
| |
sqlalchemy-migrate does not (and will not) support sqlalchemy 2.0. We
need to drop these migrations to ensure we can upgrade our sqlalchemy
version.
Change-Id: I7756e393b78296fb8dbf3ca69c759d75b816376d
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib. Most of this
is autogenerated, as described below, but there is one manual change
necessary:
nova/tests/functional/regressions/test_bug_1781286.py
We need to avoid using 'fixtures.MockPatch' since fixtures is using
'mock' (the library) under the hood and a call to 'mock.patch.stop'
found in that test will now "stop" mocks from the wrong library. We
have discussed making this configurable but the option proposed isn't
that pretty [1] so this is better.
The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:
import glob
for path in glob.glob('nova/tests/**/*.py', recursive=True):
with open(path) as fh:
lines = fh.readlines()
if 'import mock\n' not in lines:
continue
import_group_found = False
create_first_party_group = False
for num, line in enumerate(lines):
line = line.strip()
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
for lib in (
'ddt', 'six', 'webob', 'fixtures', 'testtools'
'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
):
if lib in tokens[1]:
create_first_party_group = True
break
if create_first_party_group:
break
import_group_found = True
if not import_group_found:
continue
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
if tokens[1] > 'unittest':
break
elif tokens[1] == 'unittest' and (
len(tokens) == 2 or tokens[4] > 'mock'
):
break
elif not line:
break
if create_first_party_group:
lines.insert(num, 'from unittest import mock\n\n')
else:
lines.insert(num, 'from unittest import mock\n')
del lines[lines.index('import mock\n')]
with open(path, 'w+') as fh:
fh.writelines(lines)
Note that we cannot remove mock from our requirements files yet due to
importing pypowervm unit test code in nova unit tests. This library
still uses the mock lib, and since we are importing test code and that
lib (correctly) only declares mock in its test-requirements.txt, mock
would not otherwise be installed and would cause errors while loading
nova unit test code.
[1] https://github.com/testing-cabal/fixtures/pull/49
Change-Id: Id5b04cf2f6ca24af8e366d23f15cf0e5cac8e1cc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We discovered that two unit test cases added in
I0647bb8545c1464b521a1d866cf5ee674aea2eae cause errors like
oslo_db.sqlalchemy.enginefacade.AlreadyStartedError:
this TransactionFactory is already started
when the db tests run selectively with tox -e py38 nova.tests.unit.db
but does not cause errors if the whole unit test suit is run.
This error happened because our db code uses two global transaction
factory, one for the api DB and one for the main DB. There was a global
flag SESSION_CONFIGURED in our Database fixture that guarded against
double initialization of the factory. But the faulty test cases in
question do not use our Database fixture but use the
OpportunisticDBTestMixin from oslo_db. Obviously that fixture does not
know about our SESSION_CONFIGURED global. So if one of the offending
test case ran first in an executor then that initialized the
transaction factory globally and a later test that uses our Database
fixture tried to configure it again leading to the error. For some
unknown reason if these tests were run in the opposite order the faulty
re-initialization did not happen. Probably the OpportunisticDBTestMixin
was able to prevent that.
A previous patch already removed the global SESSION_CONFIGURED flag
from our fixture and replaced it with a per DB specific patch_factory
calls that allow resetting the state of the factory at the end of each
test case. This would already solve the current test case issue as only
our offending test cases would initialize the global factory without
cleanup and we have one test case per DB. So there would be no
interference. However if in the future we add similar test cases then
those can still interfere through the global factory.
So this patch fixes the two offending test case. Also it extends the
DatabasePoisonFixture used for the NoDbTestCase tesst. The poison now
detects if the test case starts any transaction factory.
This poison caught another offender test case,
test_db_sync_with_special_symbols_in_connection_string, that was marked
NoDb but actually using the database. It is now changed to declare
itself as a test that sets up the DB manually and also it is changed to
use the Database fixture instead of touching the global factory
directly.
Closes-Bug: #1948963
Change-Id: Id96f1795034490c13125ebbab49b029fb96af1c7
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
These fields were never used in the API database. They can be removed
now, some years after originally intended.
Change-Id: I781875022d37d2c0626347f42c87707a29a9ab21
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |\ \
| |/ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We have a policy of removing fields from SQLAlchemy models at least one
cycle before we remove the underlying database columns. This can result
in a discrepancy between the state that our newfangled database
migration tool, alembic, sees and what's actually in the database. We
were ignoring these removed fields (and one foreign key constraint) in
two different locations for both databases: as part of the alembic
configuration and as part of the tests we have to ensure our migrations
are in sync with our models (note: the tests actually use the alembic
mechanism to detect the changes [1]). De-duplicate these.
[1] https://github.com/sqlalchemy/alembic/issues/724#issuecomment-672081357
Change-Id: I978b4e44cf7f522a70cc5ca76e6d6f1a985d5469
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We're seeing quite a few timeout failures on the following tests in
'nova.tests.unit.db.main.test_migrations':
- TestModelsLegacySyncMySQL.test_models_sync
- TestMigrationsWalkMySQL.test_walk_versions
- TestModelsSyncMySQL.test_innodb_tables
- TestModelsSyncMySQL.test_models_sync
Evidently MySQL is particularly affected here. Test run times are slow
even on a relatively powered machine like my localhost (Lenovo T460s w/
Intel Core i7-6600U CPU + 20G RAM) and the CI machines are only making
matters worse. Local experiments with alternative MySQL libraries, such
'mysqlclient', did not yield any improvements in performance so we must
simply live with this for now. Do so by setting 'TIMEOUT_SCALING_FACTOR'
for these tests to 4, meaning these tests will now get a whopping 640
seconds (or over 10 minutes) to execute (we set OS_TEST_TIMEOUT to 160
in 'tox.ini'). We set this for both main and API DB migrations, even
though only the former is currently exhibiting issues, to head off
future problems. An alternative to this would be to override the timeout
on a test-by-test basis, as Cinder has done [1], but that seems more
complicated for no good reason. Yet another alternative would be to
reintroduce the serialization of these tests first introduced in change
I6ce930fa86c82da1008089791942b1fff7d04c18, but that is left until later
in the hopes that simply increasing the timeout will resolve the issues.
[1] https://github.com/openstack/cinder/blob/19.0.0/cinder/tests/unit/db/test_migrations.py
Change-Id: I82b9a064d77251945ff1ae99d7049f367ddde92e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
| |
Change-Id: I2e970ec4a185cd28611855004b6d9786bf2ce217
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
For some reason, 'alembic.script.ScriptDirectory.walk_revisions' gives
us the revisions in reverse chronological order, meaning we need to
invert this in tests.
In addition, there were some typos and we didn't include boilerplate to
allow users to write pre- and post-upgrade checks as part of the test.
There was also an unnecessary '__init__' file along with some unused
test helpers left over. Address all issues.
Change-Id: I52a139bdf43b11268999ca0ebb940e09b567b739
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When nova switched to alembic implementation was added to nova-manage db
version CLI to query the current db revision from alembic. However
multiple mistake was made.
The code called alembic_api.current[1] with an Engine object while that
call expects a Config object instead. This leads to but/1943436.
Also the same code expected that this call returns the revision. But
that call just prints the revision to the standard output instead.
So the implementations has been change from calling the alembic command
API which is mostly created for CLI consumption to
MigrationContext.get_current_revision() call that is intended to be used
as a python API instead.
[1] https://alembic.sqlalchemy.org/en/latest/api/commands.html#alembic.command.current
Co-Authored-By: Sean Mooney <smooney@redhat.com>
Closes-Bug: #1943436
Change-Id: I9fa7c03310c5bdb82e9a9c39727edb12eeae77f0
|
| |
|
|
|
|
|
|
|
|
| |
The added db tests reproduces the bug that
$ nova-manage db version fails with
AttributeError: 'Engine' object has no attribute 'get_main_option'
If the db is managed by alembic.
Change-Id: I0647bb8545c1464b521a1d866cf5ee674aea2eae
Related-Bug: #1943436
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Let's take advantage of the features that alembic provides [1]. We use
this opportunity to clean up how we do our base model creation and
remove the downgrade section from the migration script template, since
we don't support downgrades.
[1] https://alembic.sqlalchemy.org/en/latest/autogenerate.html
Change-Id: I18846a5c7557db45bb63b97c7e8be5c4367e4547
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This looks more complicated than it is, but it's really quite simple.
Essentially we have to deal with two possible configurations:
- For existing deployments, the DB sync operation should apply any
outstanding sqlalchemy-migrate-based migrations, dummy apply the
initial alembic migration, and then apply any additional alembic-based
migrations requested (or any available, if no version is specified).
- For new deployments, the DB sync operation should apply the initial
alembic migration and any additional alembic-based migrations
requested (or any available, if no version is specified). No
sqlalchemy-migrate-based migrations will ever be applied.
While we continue to allow users to request a specific database
migration version to upgrade to, we *do not* allow them to request a
sqlalchemy-migrate-based migration version. There's no good reason to do
this - the deployment won't run with an out-of-date DB schema (something
that's also true of the alembic migration, fwiw) - and we want to get
people off of sqlalchemy-migrate as fast as possible. A change in a
future release can remove the sqlalchemy-migrate-based migrations once
we're sure that they'll have upgraded to a release including all of the
sqlalchemy-migrated-based migrations (so Wallaby).
Tests are modified to validate the sanity of these operations. They're
mostly trivial changes, but we do need to do some funky things to ensure
that (a) we don't use logger configuration from 'alembic.ini' that will
mess with our existing logger configuration and (b) we re-use connection
objects as necessary to allow us to run tests against in-memory
databases, where a different connection would actually mean a different
database. We also can't rely on 'WalkVersionsMixin' from oslo.db since
that only supports sqlalchemy-migrate [1]. We instead must re-invent the
wheel here somewhat.
[1] https://github.com/openstack/oslo.db/blob/10.0.0/oslo_db/sqlalchemy/test_migrations.py#L42-L44
Change-Id: I850af601f81bd5d2ecc029682ae10d3a07c936ce
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
We're going to be extensively reworking these to handle the alembic
switchover in a future change. Ahead of this rework, rework things
somewhat to simplify things and remove unnecessary noise.
Change-Id: Ie9d00e25b7e99104155466060a2b6f8a884a5e0a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
|
|
Introduce a new 'nova.db.api.api' module to hold API database-specific
helpers, plus a generic 'nova.db.utils' module to hold code suitable for
both main and API databases. This highlights a level of complexity
around connection management that is present for the main database but
not for the API database. This is because we need to handle the
complexity of cells for the former but not the latter.
Change-Id: Ia5304c552ce552ae3c5223a2bfb3a9cd543ec57c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
|
