diff options
| author | whoami-rajat <rajatdhasmana@gmail.com> | 2022-04-18 21:43:14 +0530 |
|---|---|---|
| committer | Bence Romsics <bence.romsics@gmail.com> | 2022-04-25 14:56:39 +0200 |
| commit | a4ba0c4fded13190f9161acedf6240f025718a77 (patch) | |
| tree | 6321a08a7c2f3026d3c9d4b94c5b4bde72ee3f76 | |
| parent | ea5139be9a76ca8c2e60d12d8dd776bff39c4579 (diff) | |
| download | glance_store-2.7.1.tar.gz | |
When we try to simultaneously attach same volume multiple times
(multiattach), there are multiple attachments created, suppose
attachA and attachB. If attachA marks the volume "reserved" then
attachB can't proceed until the volume is in "in-use" or "available"
state. We retry until we reach any of these states for which we use
the retrying library.
The retrying library defaults to 1 second retry[1] (5 times) which causes
the original failure as the volume takes time to transition between
"reserved" -> "in-use" state. This patch corrects it by adding an
exponential retry time and max exponential retry i.e.
1: 2 ** 1 = 2 seconds
2: 2 ** 2 = 4 seconds
3: 2 ** 3 = 8 seconds
4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)
[1] https://github.com/rholder/retrying/blob/master/retrying.py#L84
Closes-Bug: #1969373
Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea
(cherry picked from commit ba4af147fb93d522f8cf396098891f4cbe7b30fb)
(cherry picked from commit c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d)
| -rw-r--r-- | glance_store/common/cinder_utils.py | 4 | ||||
| -rw-r--r-- | releasenotes/notes/fix-interval-in-retries-471155ff34d9f0e9.yaml | 7 |
2 files changed, 10 insertions, 1 deletions
diff --git a/glance_store/common/cinder_utils.py b/glance_store/common/cinder_utils.py index b3739a8..b14aa23 100644 --- a/glance_store/common/cinder_utils.py +++ b/glance_store/common/cinder_utils.py @@ -71,7 +71,9 @@ class API(object): client.volumes.delete(volume_id) @retrying.retry(stop_max_attempt_number=5, - retry_on_exception=_retry_on_bad_request) + retry_on_exception=_retry_on_bad_request, + wait_exponential_multiplier=1000, + wait_exponential_max=10000) @handle_exceptions def attachment_create(self, client, volume_id, connector=None, mountpoint=None, mode=None): diff --git a/releasenotes/notes/fix-interval-in-retries-471155ff34d9f0e9.yaml b/releasenotes/notes/fix-interval-in-retries-471155ff34d9f0e9.yaml new file mode 100644 index 0000000..00f28da --- /dev/null +++ b/releasenotes/notes/fix-interval-in-retries-471155ff34d9f0e9.yaml @@ -0,0 +1,7 @@ +--- +fixes: + - | + `Bug #1969373 <https://bugs.launchpad.net/glance-store/+bug/1969373>`_: + Cinder Driver: Correct the retry interval from fixed 1 second to + exponential backoff for attaching a volume during image create/save + operation. |