summaryrefslogtreecommitdiff
path: root/t
Commit message (Collapse)AuthorAgeFilesLines
* SQS: avoid excessive GetQueueURL calls by using cached queue url (#1621)Tom Sparrow2023-04-191-0/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix #1618: avoid re-fetching queue URL when we already have it `_get_from_sqs` was unnecessarily calling `get_queue_url` every time even though the only place which calls `_get_from_sqs` (that is `_get_async`) actually already knows the queue URL. This change avoids hundreds of `GetQueueUrl` AWS API calls per hour when using this SQS backend with celery. Also `connection` is set by the one-and-only caller (and `queue` is actually the queue name string now anyway so couldn't ever have `.connection`) so remove the None default and unused fallback code. * Clarify that `_new_queue` returns the queue URL It seems that prior to 129a9e4ed05b it returned a queue object but this is no longer the case so update comments variable names accordingly to make it clearer. Also remove the incorrect fallback which cannot be correct any more given the return value has to be the queue URL which must be a string. * Unit test coverage for SQS async codepath This key code path (which as far as I can see is the main route when using celery with SQS) was missing test coverage. This test adds coverage for: `_get_bulk_async` -> `_get_async` -> `_get_from_sqs`
* fix mongodb transport obsolete calls (#1694)Francis Charette-Migneault2023-04-181-15/+59
| | | | | * fix mongodb transport obsolete calls + add test mock specs to check valid methods * fix linting
* fix: handle keyerror in azureservicebus transport when message is not found ↵Ismael Jiménez Sánchez2023-04-111-0/+86
| | | | | | | | | in qos and perform basic_ack (#1691) * fix: handle keyerror when message is not found in qos and perform basic_ack * fix: added tests for basic_ack * fix: limit line length
* Deprecate pytz and use zoneinfo (#1680)Maxwell Muoto2023-04-091-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Main * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix * Trigger Build * Fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix * Fix * noqas * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused noqa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * re-add import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use pytest-freezer (#1683) * Main * Trigger Build * Fixes * remove * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * lint * Lint --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Adapt the mock to correctly mock the behaviors as implemented on Python ↵Jason R. Coombs2023-03-121-2/+6
| | | | 3.10. Ref #1663.
* refactor: Refactor utils/json (#1659)Serhii Tereshchenko2023-03-021-20/+14
| | | | | | | | | | | | | | | * refactor: Refactor utils/json * Update kombu/utils/json.py * Update kombu/utils/json.py * chore: Use older syntax (no walrus) * chore: Update doscstrings * chore: Fix pydocstyle complaints * chore: Restore previous docstring
* azure service bus: add type annotations and use cached propertyJason Barnett2023-01-051-2/+23
|
* add managed identity support to azure storage queue (#1631)Jason Barnett2022-12-271-0/+31
| | | | | * add managed identity support to azure storage queue * flake8 fixes
* Allowing `Connection.ensure()` to retry on specific exceptions given by ↵Tomer Nosrati2022-12-181-0/+37
| | | | | | | | | policy (#1629) * Added unit test: test_ensure_retry_errors_is_not_looping_infinitely() * Added unit test: test_ensure_retry_errors_is_limited_by_max_retries() * Added retry_errors arg to Connection.ensure() to allow applying retry policy for specific errors additionally
* Allow azurestoragequeues transport to be used with Azurite emulator in ↵David Bossanyi2022-10-161-0/+21
| | | | | | | | | | | | | | | docker-compose (#1611) * Parse credential as a dict when using Azurite emulator This more flexible credential allows the use of Azurite for integration testing in local docker-compose configurations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix some lint errors Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix errors from flake8 lintMarti Raudsepp2022-10-124-7/+7
| | | | | Python's `del` is a statement, not a function, and doesn't need parenthesis.
* Add separate transport option for retry loop timeout (#1599)Marti Raudsepp2022-10-122-1/+45
| | | | | | | | | | | | | | | | | | | | | * Add separate transport option for retry loop timeout This only applies when using `Connect.default_channel`. Before this change, the retry loop timeout was set equal to TCP connect timeout (`connect_timeout`), meaning when first connection attempt timeouted, no retry would be attempted. Now if a new transport option `connect_total_timeout` is provided, this overrides `connect_timeout` for the retry loop (but not for TCP connect). * Add tests * Fix isort * Rename to connect_retry_timeout * Reformat * connect_retry_timeout -> connect_retries_timeout * Fix flake8
* Solve Kombu filesystem transport not thread safekarajan10012022-09-231-0/+68
| | | | | | partly fix: #398 1. add exclusive lock during the whole exchange file update. 2. add some unit test for file lock
* Make JSONEncoder keep the same type for date/datetime.Manuel Vázquez Acosta2022-09-211-2/+2
| | | | | | | Otherwise Celery jobs start to get `datetime` in place of `date` and that could lead to errors. See https://github.com/celery/celery/issues/7754, related PR #1515.
* Revert "Solve Kombu filesystem transport not thread safe (#1593)" (#1595)Gao2022-09-091-111/+1
| | | This reverts commit 8699920e050727d385a6d5a19c939e55a86688d6.
* Solve Kombu filesystem transport not thread safe (#1593)Gao2022-09-071-1/+111
| | | | | | | | | | | | | | * Solve Kombu filesystem transport not thread safe fix: #398 Currently only write lock used in msg/exchange file written. Cause reading in other thread got some incomplete result. 1. Add timeout for the lock acquire. 2. Add Share locks when reading message from filesystem. 3. Add a unit test for the `lock` and `unlock` 4. Add a unit test to test the lock during message processing. * Replace deprecated function.
* Fix incompatibility with redis in disconnect() (#1589)Shalabh Chaturvedi2022-08-251-0/+16
| | | | | * Accept *args in disconnect() * Add test for redis connection timeout error
* hub: tick delay fix (#1587)Shahar Lev2022-08-241-1/+18
| | | | | | | | | | | | | | | | | | | | | * hub: tick delay fix todo and timer callbacks can perform actions that require a tick callback to be executed right away without polling. the current order can cause issues when using single worker with no prefetch (acks late). related issue in celery: https://github.com/celery/celery/issues/7718 * add unit test for hub delay fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Avoid losing type of UUID when serializing/deserializing (#1575)Carlos Gottberg2022-07-171-3/+12
| | | | | | | | | | | | | | | | | | | | | * Avoid losing type of UUID when serializing/deserializing Serializing UUIDs as strs and deserializing them as strs can lead to somewhat obscure bugs such as the one that led to the creation of this PR in django-cacheback https://github.com/codeinthehole/django-cacheback/pull/100 which some would call unexpected behaviour. After all, an UUID is altogether a different type from strs and if bytes got their own serializer/deserializer for UUID the same logic should apply. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update documentation for JSON serialization Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add WATCH to prefixed complex commandsDan Cecile2022-07-141-0/+31
|
* Add support to SQS DelaySeconds (#1567)eloranger2022-07-121-0/+24
| | | | | | | | | | | | | * Add DelaySeconds to kwargs * Add test * add default value for DelaySeconds * Fix tests and add check for properties * Fix flake8 style issue Co-authored-by: Edmund Lam <2623895+edmundlam@users.noreply.github.com>
* Datetime serialization and deserialization fixed (#1515)dobosevych2022-05-311-5/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Datetime serialization and deserialization fixed * Unit test fixed * Unit test fixed * Fixed pylint * Added Undocumented Autodoc Modules * Update kombu/utils/json.py Co-authored-by: Omer Katz <omer.katz@omerkatz.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clean and freeze now * Clean and freeze now * Clean and freeze now * Clean and freeze now Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Omer Katz <omer.katz@omerkatz.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Annotate `abstract.py` (#1522)Marcelo Trylesinski2022-05-151-60/+60
| | | | | | | | | | | | | | | | | | | * Annotate `abstract.py` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * apply pre-commit * Use quotes * Add typing_extensions as requirement * Add quotes * Add quotes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Upgrade Azure Storage Queues transport to version 12 (#1539)Jonas Miederer2022-04-231-0/+33
| | | | | | | | | | | | | | | | * updated azurestoragequeues transport for azure-storage-queues v12 + added basic tests * fixed flake8 issues * pinned azure-storage-queue lib to >= v12.0.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * azure-storage-queue>=12.2.0 Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Kafka support (#1506)CountRedClaw2022-04-211-0/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add kafka support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix checks * Fix checks * Update kombu/transport/confluentkafka.py Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> * Add customization ability * Add docs * Fix pypi integration tests * Fix pydocstyle * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup * Disable PyPy tests * Fix tests * Add docs Co-authored-by: ilbo0518 <ilia.borsuk@netcracker.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Annotate `exceptions.py` and `clocks.py` (#1526)Marcelo Trylesinski2022-04-182-9/+10
| | | | | * Annotate `exceptions.py` * Annotate `clocks.py`
* Support pymongo 4.x (#1536)Jakub Pieńkowski2022-04-183-42/+170
| | | | | | | | | | | * Support pymongo 4.x * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix problems detected by CI Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump pyupgrade version and add __future__.annotations importMarcelo Trylesinski2022-04-1565-14/+142
|
* chore: Update `__exit__` signaturesSondre Lillebø Gundersen2022-04-129-33/+62
| | | | Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com>
* Annotate `utils/uuid.py` and `utils/text.py` (#1514)Marcelo Trylesinski2022-04-121-2/+2
| | | | | | | * Annotate `utils/uuid.py` and `utils/text.py` * Use callable without any parameter * Update kombu/utils/text.py
* Added possibility to serialize and deserialize binary messages in json (#1516)dobosevych2022-04-121-0/+12
| | | | | | | | | | | | | | | | | * Added possibility to serialize and deserialize binary messages in json * Flake8 fixed * Hypothesis added to improve test range. Fixed issue b'\x80' serialization. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added docstring * Fixed pylint Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Protect set of ready tasks by lock to avoid concurrent updates. (#1489)Oliver Nemček2022-03-241-0/+17
| | | | | | | | | | When there is no locking then there is a possibility that multiple threads manipulate with the same object at the same time. The issue is manifested as: ``` RuntimeError: Set changed size during iteration ``` See: https://github.com/celery/celery/issues/7162
* Add fanout to filesystem (#1499)Gao2022-03-151-0/+94
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Create a folder for each queue when using filesystem transport and add fanout support * clean up unused variables * Add fanout support to filesystem transport filesystem transport lacks of fanout support. 1. Add fanout support to filesystem transport. 2. Add a unit test for it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove the refactoring work and make the test passed 1. Remove all of refactoring work 2. make the test pass * Use pathlib for some Path operation * Some reviewed changes Co-authored-by: Yuriy Halytskyy <y.halytskyy@auckland.ac.nz> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add an option to not base64-encode SQS messages.Shane Hathaway2022-03-041-4/+4
| | | | | Also simplify the base64 decoding logic so that we don't have to run base64 decoding twice for every message.
* Added global_keyprefix support for pubsub clients (#1495)vinay karanam2022-02-281-0/+20
| | | | | * Added global_keyprefix support for pubsub clients * Added test cases
* Warn about missing hostname only when default one is available (#1488)Michael Manganiello2022-02-021-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Warn about missing hostname only when default one is available The `No hostname was supplied` warning is affecting projects that use AWS SQS (as detailed in #1357), as a hostname is not required when setting up the broker URL. Instead, the official documentation [0] specifies that the valid broker URL formats are: * `sqs://` * `sqs://aws_access_key_id:aws_secret_access_key@` With these formats, the `kombu.utils.url.parse_url` util doesn't return a hostname, and workers end up triggering the following warning: > No hostname was supplied. Reverting to default 'None' As the SQS transport doesn't provide a default value for hostname, this diff changes the behavior to only warn the user when the hostname hasn't been supplied but a default one is being set by the default connection parameters for the defined transport. Fixes #1357. [0] https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/sqs.html#configuration * Use caplog default logging level of WARNING
* Remove use of OrderedDict in various places (#1483)Mads Jensen2022-01-301-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | * Remove use of OrderedDict in Connection.info. * Remove remnant use of collections.OrderedDict * Undo QoS._delivered move to Python standard dictionary. This requires more work to convert, due to a hack in how the dictionary is used. * Undo LRUCache.data to dict conversion. It's also problematic, and caused tests to fail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert QoS._delivered comment chagne. * Update comment Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Some small updates (#1486)Mads Jensen2022-01-301-1/+1
| | | | | * Some small updates * Update QpidException super-call to Python 3 syntax.
* Set redelivered property for Celery with Redis (#1484)Denis Kubashevskiy2022-01-281-0/+63
| | | | | | | | | * Set redelivered property for Celery with Redis Fixed setting `redelivered` value for Celery when Redis broker is used. * Add `test_do_restore_message_celery` test * Fix long line
* Fix issue #789: Async http code not allowing for proxy config (#790)John Koehl2022-01-141-1/+23
| | | | | | | * (Issue #789) Don't reset the proxy config in the async curl based http request building unless that is specified by the caller * Test curl proxy only set when explicitly specified by caller Co-authored-by: Adam Aljets <aljets@users.noreply.github.com>
* allow getting recoverable_connection_errors without an active transport (#1471)Paul Brown2021-12-303-11/+110
| | | | | | | | | | | | | | | | | | | | | * allow getting recoverable_connection_errors without an active transport * move redis transport errors to class * move consul transport errors to class * move etcd transport errors to class * remove redis.Transport._get_errors and references in tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix flake8 errors * add integration test for redis ConnectionError Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* prevent redis event loop stopping on 'consumer: Cannot connect' (#1477)Paul Brown2021-12-251-4/+9
| | | | | | | * prevent redis event loop stopping on 'consumer: Cannot connect' * fix flake8 errors * simplify event pool polling started check
* respect connection timeout (#1458)Mehdi Pourfar2021-12-241-0/+10
| | | | | * pass timeout to ensure connection function * test connection respect its timeout
* prevent event loop polling on closed redis transports (and causing leak)Paul Brown2021-12-241-0/+15
|
* reduce memory usage of Transport (#1470)Paul Brown2021-12-232-6/+33
| | | | | | | * reduce memory usage of Transport * fix flake8 errors * move channel_id login into _get_free_channel_id
* Ensure that restore is atomic in redis transport (#1444)Matus Valo2021-12-104-28/+99
| | | | | | | | | | | | | | | | | | | | | | | * Ensure that restore is atomic in redis transport * Fix wrong variable * Added integration tests * Update unittests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix integration tests * Fix flake8 * Fix queue name in int. tests * Cosmetic clean up of integration tests Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* SQS transport: detect FIFO queue properly by checking queue URL (#1450)n0061q2021-12-031-0/+53
| | | | | | | * SQS: detect FIFO queue properly * SQS: validate predefined queues Co-authored-by: Yaroslav Litus <Yaroslav.Litus@F-Secure.com>
* Added integration test of failed authentication to redisMatus Valo2021-11-061-2/+22
|
* Return empty list instead of InconsistencyError when exchange table is emptyMatus Valo2021-10-252-9/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Missing redis key containg set of queues bound to queue is caused by removal all queues from the exchange. Hence, we should not raise an exception but just return empty list of queues instead. This commit fixes e.g. case publishing message against empty exchange: import kombu conn = kombu.Connection('redis://') exchange = kombu.Exchange('name', type='direct') exchange.declare(channel=conn.default_channel) producer = conn.Producer() producer.publish( {'hello': 'world'}, exchange=exchange, routing_key='queue1' ) But it also fixes the case when last queue is unbound from exchange and after publishing to this exchange: import kombu conn = kombu.Connection('redis://') exchange = kombu.Exchange('name', type='direct') queue1 = kombu.Queue('queue1', exchange=exchange, routing_key='queue1') exchange.declare(channel=conn.default_channel) queue1 = queue1.bind(channel=conn.default_channel) queue1.declare() producer = conn.Producer() producer.publish( {'hello': 'world'}, exchange=exchange, routing_key='queue1' ) queue1.delete() producer.publish( {'hello': 'world'}, exchange=exchange, routing_key='queue1' )
* Fix test_pidbox unittests to support non-linux platforms (#1398)Matus Valo2021-10-051-5/+1
|
View Lorry Controller Status