docs: Add documentation on database migrations

Alembic does lots of new things. Provide docs for how to use this. We
also improve upgrade docs slightly, removing references to ancient
reviews that are no longer really helpful as well as calling out our N
-> N+1 constraint.

Change-Id: I3760b82ce3bd71aa0a760d7137d69dfa3f29dc1d
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

3 files changed, 286 insertions, 90 deletions

diff --git a/doc/source/reference/database-migrations.rst b/doc/source/reference/database-migrations.rst
new file mode 100644
index 0000000000..add7597e93
--- /dev/null
+++ b/doc/source/reference/database-migrations.rst
@@ -0,0 +1,187 @@
+===================
+Database migrations
+===================
+
+.. note::
+
+   This document details how to generate database migrations as part of a new
+   feature or bugfix. For info on how to apply existing database migrations,
+   refer to the documentation for the :program:`nova-manage db sync` and
+   :program:`nova-manage api_db sync` commands in :doc:`/cli/nova-manage`.
+   For info on the general upgrade process for a nova deployment, refer to
+   :doc:`/admin/upgrades`.
+
+A typical nova deployments consists of an "API" database and one or more
+cell-specific "main" databases. Occasionally these databases will require
+schema or data migrations.
+
+
+Schema migrations
+-----------------
+
+.. versionchanged:: 24.0.0 (Xena)
+
+   The database migration engine was changed from ``sqlalchemy-migrate`` to
+   ``alembic``.
+
+The `alembic`__ database migration tool is used to manage schema migrations in
+nova. The migration files and related metadata can be found in
+``nova/db/api/migrations`` (for the API database) and
+``nova/db/main/migrations`` (for the main database(s)). As discussed in
+:doc:`/admin/upgrades`, these can be run by end users using the
+:program:`nova-manage api_db sync` and :program:`nova-manage db sync` commands,
+respectively.
+
+.. __: https://alembic.sqlalchemy.org/en/latest/
+
+.. note::
+
+   There are also legacy migrations provided in the ``legacy_migrations``
+   subdirectory for both the API and main databases. These are provided to
+   facilitate upgrades from pre-Xena (24.0.0) deployments and will be removed
+   in a future release. They should not be modified or extended.
+
+The best reference for alembic is the `alembic documentation`__, but a small
+example is provided here. You can create the migration either manually or
+automatically. Manual generation might be necessary for some corner cases such
+as renamed tables but auto-generation will typically handle your issues.
+Examples of both are provided below. In both examples, we're going to
+demonstrate how you could add a new model, ``Foo``, to the main database.
+
+.. __: https://alembic.sqlalchemy.org/en/latest/
+
+.. code-block:: diff
+
+   diff --git nova/db/main/models.py nova/db/main/models.py
+   index 7eab643e14..8f70bcdaca 100644
+   --- nova/db/main/models.py
+   +++ nova/db/main/models.py
+   @@ -73,6 +73,16 @@ def MediumText():
+            sqlalchemy.dialects.mysql.MEDIUMTEXT(), 'mysql')
+
+
+   +class Foo(BASE, models.SoftDeleteMixin):
+   +    """A test-only model."""
+   +
+   +    __tablename__ = 'foo'
+   +
+   +    id = sa.Column(sa.Integer, primary_key=True)
+   +    uuid = sa.Column(sa.String(36), nullable=True)
+   +    bar = sa.Column(sa.String(255))
+   +
+   +
+    class Service(BASE, models.SoftDeleteMixin):
+        """Represents a running service on a host."""
+
+(you might not be able to apply the diff above cleanly - this is just a demo).
+
+.. rubric:: Auto-generating migration scripts
+
+In order for alembic to compare the migrations with the underlying models, it
+require a database that it can inspect and compare the models against. As such,
+we first need to create a working database. We'll bypass ``nova-manage`` for
+this and go straight to the :program:`alembic` CLI. The ``alembic.ini`` file
+provided in the ``migrations`` directories for both databases is helpfully
+configured to use an SQLite database by default (``nova.db`` for the main
+database and ``nova_api.db`` for the API database). Create this database and
+apply the current schema, as dictated by the current migration scripts:
+
+.. code-block:: bash
+
+   $ tox -e venv -- alembic -c nova/db/main/alembic.ini \
+       upgrade head
+
+Once done, you should notice the new ``nova.db`` file in the root of the repo.
+Now, let's generate the new revision:
+
+.. code-block:: bash
+
+   $ tox -e venv -- alembic -c nova/db/main/alembic.ini \
+       revision -m "Add foo model" --autogenerate
+
+This will create a new file in ``nova/db/main/migrations`` with
+``add_foo_model`` in the name including (hopefully!) the necessary changes to
+add the new ``Foo`` model. You **must** inspect this file once created, since
+there's a chance you'll be missing imports or something else which will need to
+be manually corrected. Once you've inspected this file and made any required
+changes, you can apply the migration and make sure it works:
+
+.. code-block:: bash
+
+   $ tox -e venv -- alembic -c nova/db/main/alembic.ini \
+       upgrade head
+
+.. rubric:: Manually generating migration scripts
+
+For trickier migrations or things that alembic doesn't understand, you may need
+to manually create a migration script. This is very similar to the
+auto-generation step, with the exception being that you don't need to have a
+database in place beforehand. As such, you can simply run:
+
+.. code-block:: bash
+
+   $ tox -e venv -- alembic -c nova/db/main/alembic.ini \
+       revision -m "Add foo model"
+
+As before, this will create a new file in ``nova/db/main/migrations`` with
+``add_foo_model`` in the name. You can simply modify this to make whatever
+changes are necessary. Once done, you can apply the migration and make sure it
+works:
+
+.. code-block:: bash
+
+   $ tox -e venv -- alembic -c nova/db/main/alembic.ini \
+       upgrade head
+
+
+Data migrations
+---------------
+
+As discussed in :doc:`/admin/upgrades`, online data migrations occur in two
+places:
+
+- Inline migrations that occur as part of normal run-time activity as data is
+  read in the old format and written in the new format.
+
+- Background online migrations that are performed using ``nova-manage`` to
+  complete transformations that will not occur incidentally due to normal
+  runtime activity.
+
+.. rubric:: Inline data migrations
+
+Inline data migrations are arguably the easier of the two to implement. Almost
+all of nova's database models correspond to an oslo.versionedobject (o.vo) or
+part of one. These o.vos load their data from the underlying database by
+implementing the ``obj_load_attr`` method. By modifying this method, it's
+possible to detect missing changes to the data - for example, a missing field -
+modify the data, save it back to the database, and finally return an object
+with the newly updated data. Change I6cd206542fdd28f3ef551dcc727f4cb35a53f6a3
+provides a fully worked example of this approach.
+
+The main advantage of these is that they are completely transparent to the
+operator who does not have to take any additional steps to upgrade their
+deployment: the database updates should happen at runtime as data is pulled
+from the database. The main disadvantage of this approach is that some
+records may not be frequently pulled from the database, meaning they never have
+a chance to get updated. This can prevent the eventual removal of the inline
+migration in a future release. To avoid this issue, you should inspect the
+object to see if it's something that will be loaded as part of a standard
+runtime operation - for example, on startup or as part of a background task -
+and if necessary add a blocking online migration in a later release to catch
+and migrate the laggards.
+
+.. rubric:: Online data migrations
+
+Unlike inline data migrations, online data migrations require operator
+involvement. They are run using the ``nova-manage db online_data_migrations``
+command which, as noted in :doc:`/cli/nova-manage`, this should be run straight
+after upgrading to a new release once the database schema migrations have been
+applied and the code updated. Online migrations can be blocking, in that it
+will be necessary to apply given migrations while running N code before
+upgrading to N+1. Change I44919422c48570f2647f2325ff895255fc2adf27 provides a
+fully worked example of this approach.
+
+The advantages and disadvantages of this approach are the inverse of those of
+the inline data migrations approach. While they can be used to ensure an data
+migration is actually applied, they require operator involvement and can
+prevent upgrades until fully applied.
diff --git a/doc/source/reference/index.rst b/doc/source/reference/index.rst
index 0b6531ae5e..6b397f5f5c 100644
--- a/doc/source/reference/index.rst
+++ b/doc/source/reference/index.rst
@@ -30,6 +30,8 @@ The following is a dive into some of the internals in nova.
   ``ComputeDriver.update_provider_tree`` method.
 * :doc:`/reference/upgrade-checks`: A guide to writing automated upgrade
   checks.
+* :doc:`/reference/database-migrations`: A guide to writing database
+  migrations, be they online or offline.
 * :doc:`/reference/conductor`
 
 .. todo:: Need something about versioned objects and how they fit in with
@@ -56,6 +58,7 @@ The following is a dive into some of the internals in nova.
    vm-states
    threading
    notifications
+   database-migrations
    update-provider-tree
    upgrade-checks
    conductor
diff --git a/doc/source/reference/upgrade-checks.rst b/doc/source/reference/upgrade-checks.rst
index 8977cacbf1..c84ab26337 100644
--- a/doc/source/reference/upgrade-checks.rst
+++ b/doc/source/reference/upgrade-checks.rst
@@ -2,6 +2,14 @@
 Upgrade checks
 ==============
 
+.. note::
+
+   This document details how to generate upgrade checks as part of a new
+   feature or bugfix. For info on how to apply existing upgrade checks, refer
+   to the documentation for the :program:`nova-status` command in
+   :doc:`/cli/nova-status`. For info on the general upgrade process for a nova
+   deployment, refer to :doc:`/admin/upgrades`.
+
 Nova provides automated :ref:`upgrade check tooling <nova-status-checks>` to
 assist deployment tools in verifying critical parts of the deployment,
 especially when it comes to major changes during upgrades that require operator
@@ -10,15 +18,16 @@ intervention.
 This guide covers the background on nova's upgrade check tooling, how it is
 used, and what to look for in writing new checks.
 
+
 Background
-==========
+----------
 
 Nova has historically supported offline database schema migrations
-(``nova-manage db sync``) and :ref:`online data migrations <data-migrations>`
-during upgrades.
-
-The ``nova-status upgrade check`` command was introduced in the 15.0.0 Ocata
-release to aid in the verification of two major required changes in that
+(:program:`nova-manage db sync` and :program:`nova-manage api_db sync`) and
+online data migrations (:program:`nova-manage db online_data_migrations`)
+during upgrades, as discussed in :doc:`/reference/database-migrations`.
+The :program:`nova-status upgrade check` command was introduced in the 15.0.0
+(Ocata) release to aid in the verification of two major required changes in that
 release, namely Placement and Cells v2.
 
 Integration with the Placement service and deploying Cells v2 was optional
@@ -36,8 +45,9 @@ Reference the `Ocata changes`_ for implementation details.
 
 .. _Ocata changes: https://review.opendev.org/#/q/topic:bp/resource-providers-scheduler-db-filters+status:merged+file:%255Enova/cmd/status.py
 
+
 Guidelines
-==========
+----------
 
 * The checks should be able to run within a virtual environment or container.
   All that is required is a full configuration file, similar to running other
@@ -58,9 +68,9 @@ Guidelines
   host.
 
 * Checks are typically meant to be run before re-starting and upgrading to new
-  service code, which is how `grenade uses them`_, but they can also be run
+  service code, which is how `grenade uses them`__, but they can also be run
   as a :ref:`post-install verify step <verify-install-nova-status>` which is
-  how `openstack-ansible`_ also uses them. The high-level set of upgrade steps
+  how `openstack-ansible`__ also uses them. The high-level set of upgrade steps
   for upgrading nova in grenade is:
 
   * Install new code
@@ -70,6 +80,9 @@ Guidelines
   * Run the upgrade check (``nova-status upgrade check``)
   * Restart services with new code
 
+  .. __: https://github.com/openstack-dev/grenade/blob/dc7f4a4ba/projects/60_nova/upgrade.sh#L96
+  .. __: https://review.opendev.org/#/c/575125/
+
 * Checks must be idempotent so they can be run repeatedly and the results are
   always based on the latest data. This allows an operator to run the checks,
   fix any issues reported, and then iterate until the status check no longer
@@ -81,26 +94,27 @@ Guidelines
   should be a corresponding ``request_specs`` table entry in the ``nova_api``
   database. A ``nova-manage db online_data_migrations`` routine was added in
   the Newton release to back-fill request specs for existing instances, and
-  `in Rocky`_ an upgrade check was added to make sure all non-deleted instances
-  have a request spec so compatibility code can be removed in Stein. In older
-  releases of nova we would have added a `blocker migration`_ as part of the
-  database schema migrations to make sure the online data migrations had been
-  completed before the upgrade could proceed.
+  `in Rocky`__ an upgrade check was added to make sure all non-deleted
+  instances have a request spec so compatibility code can be removed in Stein.
+  In older releases of nova we would have added a `blocker migration`__ as part
+  of the database schema migrations to make sure the online data migrations had
+  been completed before the upgrade could proceed.
+
+  .. note::
+
+     Usage of ``nova-status upgrade check`` does not preclude the need
+     for blocker migrations within a given database, but in the case of
+     request specs the check spans multiple databases and was a better
+     fit for the ``nova-status`` tooling.
 
-  .. note:: Usage of ``nova-status upgrade check`` does not preclude the need
-            for blocker migrations within a given database, but in the case of
-            request specs the check spans multiple databases and was a better
-            fit for the nova-status tooling.
+  .. __: https://review.opendev.org/#/c/581813/
+  .. __: https://review.opendev.org/#/c/289450/
 
 * All checks should have an accompanying upgrade release note.
 
-.. _grenade uses them: https://github.com/openstack-dev/grenade/blob/dc7f4a4ba/projects/60_nova/upgrade.sh#L96
-.. _openstack-ansible: https://review.opendev.org/#/c/575125/
-.. _in Rocky: https://review.opendev.org/#/c/581813/
-.. _blocker migration: https://review.opendev.org/#/c/289450/
 
 Structure
-=========
+---------
 
 There is no graph logic for checks, meaning each check is meant to be run
 independently of other checks in the same set. For example, a project could
@@ -167,94 +181,86 @@ The results feed into a standard output for the checks:
   |          service catalog.                          |
   +----------------------------------------------------+
 
-.. note:: Long-term the framework for upgrade checks will come from the
-          `oslo.upgradecheck library`_.
-
 .. _initial change: https://review.opendev.org/#/c/411517/
 .. _cells v2 check: https://review.opendev.org/#/c/411525/
 .. _placement check: https://review.opendev.org/#/c/413250/
-.. _oslo.upgradecheck library: http://opendev.org/openstack/oslo.upgradecheck/
 
-Other
-=====
 
-Documentation
--------------
+FAQs
+----
+
+- How is the ``nova-status`` upgrade script packaged and deployed?
 
-Each check should be documented in the
-:ref:`history section <nova-status-checks>` of the CLI guide and have a
-release note. This is important since the checks can be run in an isolated
-environment apart from the actual deployed version of the code and since the
-checks should be idempotent, the history / change log is good for knowing
-what is being validated.
+  There is a ``console_scripts`` entry for ``nova-status`` in the ``setup.cfg``
+  file.
 
-Backports
----------
+- Why are there multiple parts to the command structure, i.e. "upgrade" and
+  "check"?
 
-Sometimes upgrade checks can be backported to aid in pre-empting bugs on
-stable branches. For example, a check was added for `bug 1759316`_ in Rocky
-which was also backported to stable/queens in case anyone upgrading from Pike
-to Queens would hit the same issue. Backportable checks are generally only
-made for latent bugs since someone who has already passed checks and upgraded
-to a given stable branch should not start failing after a patch release on that
-same branch. For this reason, any check being backported should have a release
-note with it.
+  This is an artifact of how the ``nova-manage`` command is structured which
+  has categories of sub-commands, like ``nova-manage db`` is a sub-category
+  made up of other sub-commands like ``nova-manage db sync``. The
+  ``nova-status upgrade check`` command was written in the same way for
+  consistency and extensibility if other sub-commands need to be added later.
 
-.. _bug 1759316: https://bugs.launchpad.net/nova/+bug/1759316
+- Why is the upgrade check command not part of the standard python-\*client
+  CLIs?
 
-Other projects
---------------
+  The ``nova-status`` command was modeled after the ``nova-manage`` command
+  which is meant to be admin-only and has direct access to the database,
+  unlike other CLI packages like python-novaclient which requires a token
+  and communicates with nova over the REST API. Because of this, it is also
+  possible to write commands in ``nova-manage`` and ``nova-status`` that can
+  work while the API service is down for maintenance.
 
-A community-wide `goal for the Stein release`_ is adding the same type of
-``$PROJECT-status upgrade check`` tooling to other projects to ease in
-upgrading OpenStack across the board. So while the guidelines in this document
-are primarily specific to nova, they should apply generically to other projects
-wishing to incorporate the same tooling.
+- How should the checks be documented?
 
-.. _goal for the Stein release: https://governance.openstack.org/tc/goals/stein/upgrade-checkers.html
+  Each check should be documented in the :ref:`history section
+  <nova-status-checks>` of the CLI guide and have a release note. This is
+  important since the checks can be run in an isolated environment apart from
+  the actual deployed version of the code and since the checks should be
+  idempotent, the history / change log is good for knowing what is being
+  validated.
 
-FAQs
-----
+- Do other projects support upgrade checks?
 
-#. How is the nova-status upgrade script packaged and deployed?
+  A community-wide `goal for the Stein release`__ is adding the same type of
+  ``$PROJECT-status upgrade check`` tooling to other projects to ease in
+  upgrading OpenStack across the board. So while the guidelines in this
+  document are primarily specific to nova, they should apply generically to
+  other projects wishing to incorporate the same tooling.
 
-   There is a ``console_scripts`` entry for ``nova-status`` in the
-   ``setup.cfg`` file.
+  .. __: https://governance.openstack.org/tc/goals/stein/upgrade-checkers.html
 
-#. Why are there multiple parts to the command structure, i.e. "upgrade" and
-   "check"?
+- Where should the documentation live for projects other than nova?
 
-   This is an artifact of how the ``nova-manage`` command is structured which
-   has categories of sub-commands, like ``nova-manage db`` is a sub-category
-   made up of other sub-commands like ``nova-manage db sync``. The
-   ``nova-status upgrade check`` command was written in the same way for
-   consistency and extensibility if other sub-commands need to be added later.
+  As part of the standard OpenStack project `documentation guidelines`__ the
+  command should be documented under ``doc/source/cli`` in each project repo.
 
-#. Where should the documentation live for projects other than nova?
+  .. __: https://docs.openstack.org/doc-contrib-guide/project-guides.html
 
-   As part of the standard OpenStack project `documentation guidelines`_ the
-   command should be documented under ``doc/source/cli`` in each project repo.
+- Can upgrade checks be backported?
 
-#. Why is the upgrade check command not part of the standard python-\*client
-   CLIs?
+  Sometimes upgrade checks can be backported to aid in pre-empting bugs on
+  stable branches. For example, a check was added for `bug 1759316`__ in Rocky
+  which was also backported to stable/queens in case anyone upgrading from Pike
+  to Queens would hit the same issue. Backportable checks are generally only
+  made for latent bugs since someone who has already passed checks and upgraded
+  to a given stable branch should not start failing after a patch release on
+  that same branch. For this reason, any check being backported should have a
+  release note with it.
 
-   The ``nova-status`` command was modeled after the ``nova-manage`` command
-   which is meant to be admin-only and has direct access to the database,
-   unlike other CLI packages like python-novaclient which requires a token
-   and communicates with nova over the REST API. Because of this, it is also
-   possible to write commands in ``nova-manage`` and ``nova-status`` that can
-   work while the API service is down for maintenance.
+  .. __: https://bugs.launchpad.net/nova/+bug/1759316
 
-#. Can upgrade checks only be for N-1 to N version upgrades?
+- Can upgrade checks only be for N-1 to N version upgrades?
 
-   No, not necessarily. The upgrade checks are also an essential part of
-   `fast-forward upgrades`_ to make sure that as you roll through each release
-   performing schema (data model) updates and data migrations that you are
-   also completing all of the necessary changes. For example, if you are
-   fast forward upgrading from Ocata to Rocky, something could have been
-   added, deprecated or removed in Pike or Queens and a pre-upgrade check is
-   a way to make sure the necessary steps were taking while upgrading through
-   those releases before restarting the Rocky code at the end.
+  No, not necessarily. The upgrade checks are also an essential part of
+  `fast-forward upgrades`__ to make sure that as you roll through each release
+  performing schema (data model) updates and data migrations that you are
+  also completing all of the necessary changes. For example, if you are
+  fast forward upgrading from Ocata to Rocky, something could have been
+  added, deprecated or removed in Pike or Queens and a pre-upgrade check is
+  a way to make sure the necessary steps were taking while upgrading through
+  those releases before restarting the Rocky code at the end.
 
-.. _documentation guidelines: https://docs.openstack.org/doc-contrib-guide/project-guides.html
-.. _fast-forward upgrades: https://wiki.openstack.org/wiki/Fast_forward_upgrades
+  .. __: https://wiki.openstack.org/wiki/Fast_forward_upgrades