When getting an instance using the compute.API we call scatter_gather_single_cell() to be able to capture details when we fail to retrieve a result from a cell such as timeouts and exceptions. Currently however, we aren't logging the content of an exception if scatter_gather_single_cell() returns an exception as the result. The scatter gather method itself logs exceptions that are not of type NovaException as these represent definite unexpected errors such as database errors but NovaException handling are left for the caller to decide whether they want to log it or re-raise it and so on. It can be difficult to debug a situation where a cell is returning a NovaException result so this adds logging of the exception content in the compute API when we encounter an unexpected NovaException. The existing log message has been updated to more accurately reflect what has happened (did not respond vs exception). The assignment of the exception object in scatter gather has also been updated to not unnecessarily construct a new exception object because it (a) wasn't necessary and (b) made asserting the LOG.exception() call argument in the unit test difficult. Related-Bug: #1970087 Change-Id: Iae1c61c72be5b6017b934293e3dc079a24eeb0e7
Merge these, removing an unnecessary layer of abstraction, and place them in the new 'nova.db.main' directory. The resulting change is huge, but it's mainly the result of 's/sqlalchemy import api/main import api/' and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We also need to rework how we do the blocking of API calls since we no longer have a 'DBAPI' object that we can monkey patch as we were doing before. This is now done via a global variable that is set by the 'main' function of 'nova.cmd.compute'. The main impact of this change is that it's no longer possible to set '[database] use_db_reconnect' and have all APIs automatically wrapped in a DB retry. Seeing as this behavior is experimental, isn't applied to any of the API DB methods (which don't use oslo.db's 'DBAPI' helper), and is used explicitly in what would appear to be the critical cases (via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't seem like a huge loss. Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4 Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Replace the following items with Python 3 style code. - six.binary_type - six.integer_types - six.string_types Subsequent patches will replace other six usages. Change-Id: Ide65686cf02463045f5c32771ca949802b19636f Implements: blueprint six-removal Signed-off-by: Takashi Natsume <takanattie@gmail.com>
We have had a gate bug for a long time where occasionally the scheduler service gets into a state where many requests fail in it with CellTimeout errors. Example: Timed out waiting for response from cell <cell uuid> Through the use of much DNM patch debug logging in oslo.db, it was revealed that service child processes (workers) were sometimes starting off with already locked internal oslo.db locks. This is a known issue in python [1] where if a parent process forks a child process while a lock is held, the child will inherit the held lock which can never be acquired. The python issue is not considered a bug and the recommended way to handle it is by making use of the os.register_at_fork() in the oslo.db to reinitialize its lock. The method is new in python 3.7, so as long as we still support python 3.6, we must handle the situation outside of oslo.db. We can do this by clearing the cell cache that holds oslo.db database transaction context manager objects during service start(). This way, we get fresh oslo.db locks that are in an unlocked state when a child process begins. We can also take this opportunity to resolve part of a TODO to clear the same cell cache during service reset() (SIGHUP) since it is another case where we intended to clear it. The rest of the TODO related to periodic clearing of the cache is removed after discussion on the review, as such clearing would be unsynchronized among multiple services and for periods of time each service might have a different view of cached cells than another. Closes-Bug: #1844929 [1] https://bugs.python.org/issue6721 Change-Id: Id233f673a57461cc312e304873a41442d732c051
Framework for communication with the Cyborg API. - Standard keystoneauth1 config options for setting up authentication in the [cyborg] section of nova*.conf. - A new nova.accelerator.cyborg module containing a get_client method to return a client containing a keystoneauth1 adapter pointing to the Cyborg service with user- and service- based authentication. - Requirements updates to pull in the os-service-types release containing the 'accelerator' service type. Change-Id: Iee0766269d61948ad701911e8b0e5e24d3d6eb04 Blueprint: nova-cyborg-interaction
This reverts commit 0436a95f37df086ddc99017376cb9a312e40517a. This was meant to get us more debug details when hitting the failure but the results are not helpful [1] so revert this and the fix for the resulting regression [2]. [1] http://paste.openstack.org/show/782116/ [2] I7f9edc9a4b4930f4dce98df271888fa8082a1701 Change-Id: Iab8029f081a654278ea7dbbec79a766aea6764ae Related-Bug: #1844929
Found this error in heat gate during running grenade test job. Should not asking to provide cell_uuid if queue.get is timeout. Closes-Bug: #1847131 Change-Id: I7f9edc9a4b4930f4dce98df271888fa8082a1701
When a call to a cell in scatter_gather_cells times out we log a warning and set the did_not_respond_sentinel for that cell but it would be useful if we logged the traceback with the warning for debugging where the call is happening. Change-Id: I8f4069474a3955eea6c967d3090f2960e739224c Related-Bug: #1844929
The instance_lock_checked kwarg was added to RequestContext in [1], no longer used in Nova since [2], and deprecated with a warning in [3]. This commit removes all handling of this kwarg, which will cause using it to blow up with a TypeError in the super().__init__. [1] I1127e31d86a061a93a64ee1eb4a4d900d8bf49b5 [2] Ic18017a16c5bffee85a43db65ff17283599a27ba [3] Ib0b23a47c2e833073108af6700b16f8026631a83 Change-Id: Ie5f5a0ae018ce601c588a466399515159d8f58c0