From 2e9856023c06e8bd8866608c735705e2835aacc0 Mon Sep 17 00:00:00 2001 From: gal cohen Date: Tue, 2 Mar 2021 12:22:47 +0200 Subject: SQS back-off policy (#1301) * sqs retry policy * add test * method definition * add validation * rename policy * add kombu doc * improve docstring * add test and doc * test fix * test fixes * add doc * Update docs/reference/kombu.transport.SQS.rst Co-authored-by: Omer Katz * Update docs/reference/kombu.transport.SQS.rst Co-authored-by: Omer Katz * Update kombu/transport/SQS.py Co-authored-by: Omer Katz * Update kombu/transport/SQS.py Co-authored-by: Omer Katz * Update kombu/transport/SQS.py Co-authored-by: Omer Katz * review improvements * improvements * add improvements * rename Co-authored-by: galcohen Co-authored-by: Omer Katz Co-authored-by: Asif Saif Uddin --- docs/reference/kombu.transport.SQS.rst | 40 ++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) (limited to 'docs') diff --git a/docs/reference/kombu.transport.SQS.rst b/docs/reference/kombu.transport.SQS.rst index f9abcfae..87e11d28 100644 --- a/docs/reference/kombu.transport.SQS.rst +++ b/docs/reference/kombu.transport.SQS.rst @@ -22,3 +22,43 @@ .. autoclass:: Channel :members: :undoc-members: + +Back-off policy +------------------------ +Back-off policy is using SQS visibility timeout mechanism altering the time difference between task retries. +The number of retries is managed by SQS (specifically by the ``ApproximateReceiveCount`` message attribute) and no further action is required by the user. + +Configuring the queues and backoff policy:: + + broker_transport_options = { + 'predefined_queues': { + 'my-q': { + 'url': 'https://ap-southeast-2.queue.amazonaws.com/123456/my-q', + 'access_key_id': 'xxx', + 'secret_access_key': 'xxx', + 'backoff_policy': {1: 10, 2: 20, 3: 40, 4: 80, 5: 320, 6: 640}, + 'backoff_tasks': ['svc.tasks.tasks.task1'] + } + } + } + + +``backoff_policy`` dictionary where key is number of retries, and value is delay seconds between retries (i.e +SQS visibility timeout) +``backoff_tasks`` list of task names to apply the above policy + +The above policy: + ++-----------------------------------------+--------------------------------------------+ +| **Attempt** | **Delay** | ++-----------------------------------------+--------------------------------------------+ +| ``2nd attempt`` | 20 seconds | ++-----------------------------------------+--------------------------------------------+ +| ``3rd attempt`` | 40 seconds | ++-----------------------------------------+--------------------------------------------+ +| ``4th attempt`` | 80 seconds | ++-----------------------------------------+--------------------------------------------+ +| ``5th attempt`` | 320 seconds | ++-----------------------------------------+--------------------------------------------+ +| ``6th attempt`` | 640 seconds | ++-----------------------------------------+--------------------------------------------+ -- cgit v1.2.1