summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Use test.fixtures.version not test.conftest.version to avoid warningsfix_fixture_version_warningDana Powers2019-03-064-8/+4
|
* Drop dependency on sphinxcontrib-napoleonStanislav Levin2019-02-271-1/+0
| | | | | | | | | Since 1.3b1 (released Oct 10, 2014) Sphinx has support for NumPy and Google style docstring support via sphinx.ext.napoleon extension. The latter is already used, but sphinxcontrib-napoleon requirement still presents. Signed-off-by: Stanislav Levin <slev@altlinux.org>
* Fix test_legacy_correct_metadata_response on x86 arch (#1718)Stanislav Levin2019-02-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | The problem is that the type of required operation result is "long". ``` >>> type(278251978 & 0xffffffff) <type 'long'> ``` However, by default "format" method uses __format__(): ``` >>> (278251978 & 0xffffffff).__format__('') '278251978' ``` So, let's compare things using the same engine: ``` >>> "{!r}".format(278251978 & 0xffffffff) '278251978L' ``` Fixes: https://github.com/dpkp/kafka-python/issues/1717 Signed-off-by: Stanislav Levin <slev@altlinux.org>
* Remove unused importJeff Widman2019-01-281-1/+0
|
* Improve KafkaConsumer join group / only enable Heartbeat Thread during ↵Dana Powers2019-01-151-11/+23
| | | | stable group (#1695)
* Travis CI: 'sudo' tag is now deprecated in Travis (#1698)cclauss2019-01-131-2/+0
|
* Remove unused `skip_double_compressed_messages`Jeff Widman2019-01-132-16/+0
| | | | | | | | | | This `skip_double_compressed_messages` flag was added in https://github.com/dpkp/kafka-python/pull/755 in order to fix https://github.com/dpkp/kafka-python/issues/718. However, grep'ing through the code, it looks like it this is no longer used anywhere and doesn't do anything. So removing it.
* Timeout all unconnected conns (incl SSL) after request_timeout_msDana Powers2019-01-131-6/+8
|
* Fix `AttributeError` caused by `getattr()`Jeff Widman2019-01-071-1/+2
| | | | | | | `getattr(object, 'x', object.y)` will evaluate the default argument `object.y` regardless of whether `'x'` exists. For details see: https://stackoverflow.com/q/31443989/770425
* Use Popen.communicate() instead of Popen.wait()Brian Sang2019-01-051-9/+10
| | | | | Popen objects may deadlock when using stdout=PIPE or stderr=PIPE with Popen.wait(). Using Popen.communicate() avoids the issue.
* Fix SSL connection testing in Python 3.7Ben Weir2019-01-031-0/+7
|
* Fix response error checking in KafkaAdminClient send_to_controllerDana Powers2019-01-032-5/+15
| | | | | | | | | | | | Previously we weren't accounting for when the response tuple also has a `error_message` value. Note that in Java, the error fieldname is inconsistent: - `CreateTopicsResponse` / `CreatePartitionsResponse` uses `topic_errors` - `DeleteTopicsResponse` uses `topic_error_codes` So this updates the `CreateTopicsResponse` classes to match. The fix is a little brittle, but should suffice for now.
* #1681 add copy() in metrics() to avoid thread safety issues (#1682)Tosi Émeric2018-12-272-4/+4
|
* Bugfix: Types need identity comparisonJeff Widman2018-12-131-1/+1
| | | `isinstance()` won't work here, as the types require identity comparison.
* Bump version for developmentDana Powers2018-11-201-1/+1
|
* Release 1.4.41.4.4Dana Powers2018-11-203-5/+42
|
* Cleanup formatting, no functional changesJeff Widman2018-11-201-23/+23
| | | | Cleanup the formatting, remove parens, extraneous spaces, etc.
* Rename KafkaAdmin to KafkaAdminClientJeff Widman2018-11-206-26/+26
|
* Update kafka broker compatibility docsDana Powers2018-11-203-6/+9
|
* Bump travis test for 1.x brokers to 1.1.1Dana Powers2018-11-201-1/+1
|
* Add test resources for kafka versions 1.0.2 -> 2.0.1Dana Powers2018-11-2016-1/+941
|
* Break KafkaClient poll if closedDana Powers2018-11-201-0/+2
|
* Add protocols for {Describe,Create,Delete} AclsUlrik Johansson2018-11-191-0/+185
|
* Bugfix: Always set this_groups_coordinator_idJeff Widman2018-11-191-1/+3
|
* Various docstring / pep8 / code hygiene cleanupsJeff Widman2018-11-181-71/+86
|
* Fix describe_groupsJeff Widman2018-11-181-13/+50
| | | | | | | | | | | | | This was completely broken previously because it didn't lookup the group coordinator of the consumer group. Also added basic error handling/raising. Note: I added the `group_coordinator_id` as an optional kwarg. As best I can tell, the Java client doesn't include this and instead looks it up every time. However, if we add this, it allows the caller the flexibility to bypass the network round trip of the lookup if for some reason they already know the `group_coordinator_id`.
* Set a clear default value for `validate_only`/`include_synonyms`Jeff Widman2018-11-181-8/+8
| | | | | | | Set a clear default value for `validate_only` / `include_synonyms` Previously the kwarg defaulted to `None`, but then sent a `False` so this makes it more explicit and reduces ambiguity.
* Fix list_consumer_groups() to query all brokersJeff Widman2018-11-181-5/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | Previously, this only queried the controller. In actuality, the Kafka protocol requires that the client query all brokers in order to get the full list of consumer groups. Note: The Java code (as best I can tell) doesn't allow limiting this to specific brokers. And on the surface, this makes sense... you typically don't care about specific brokers. However, the inverse is true... consumer groups care about knowing their group coordinator so they don't have to repeatedly query to find it. In fact, a Kafka broker will only return the groups that it's a coordinator for. While this is an implementation detail that is not guaranteed by the upstream broker code, and technically should not be relied upon, I think it very unlikely to change. So monitoring scripts that fetch the offsets or describe the consumers groups of all groups in the cluster can simply issue one call per broker to identify all the coordinators, rather than having to issue one call per consumer group. For an ad-hoc script this doesn't matter, but for a monitoring script that runs every couple of minutes, this can be a big deal. I know in the situations where I will use this, this matters more to me than the risk of the interface unexpectedly breaking.
* Add list_consumer_group_offsets()Jeff Widman2018-11-182-1/+77
| | | | | | | | | | | | | | | | | | | | | Support fetching the offsets of a consumer group. Note: As far as I can tell (the Java code is a little inscrutable), the Java AdminClient doesn't allow specifying the `coordinator_id` or the `partitions`. But I decided to include them because they provide a lot of additional flexibility: 1. allowing users to specify the partitions allows this method to be used even for older brokers that don't support the OffsetFetchRequest_v2 2. allowing users to specify the coordinator ID gives them a way to bypass a network round trip. This method will frequently be used for monitoring, and if you've got 1,000 consumer groups that are being monitored once a minute, that's ~1.5M requests a day that are unnecessarily duplicated as the coordinator doesn't change unless there's an error.
* Fix send to controllerJeff Widman2018-11-181-44/+89
| | | | | | | | | The controller send error handling was completely broken. It also pinned the metadata version unnecessarily. Additionally, several of the methods were sending to the controller but either that was unnecessary, or just plain wrong. So updated following the pattern of the Java Admin client.
* Add group coordinator lookupJeff Widman2018-11-181-0/+40
| | | | | | | | | | | | | | | | | | | | | | | We need a way to send a request to the group coordinator. I spent a day and a half trying to implement a `_send_request_to_group_coordinator()` that included: 1. caching the value of the group coordinator so that it wouldn't have to be repeatedly looked up on every call. This is particularly important because the `list_consumer_groups()`, `list_consumer_group_offsets()`, and `describe_consumer_groups()` will frequently be used by monitoring scripts. I know across the production clusters that I support, using a cached value will save ~1M calls per day. 2. clean and consistent error handling. This is difficult because the responses are inconsistent about error codes. Some have a top-level error code, some bury it within the description of the actual item. 3. Avoiding tight coupling between this method and the request/response classes... the custom parsing logic for errors etc, given that it's non-standard, should live in the callers, not here. So finally I gave up and just went with this simpler solution and made it so the callers can optionally bypass this if they somehow already know the group coordinator.
* Remove support for api versions as strings from KafkaAdminJeff Widman2018-11-181-10/+0
| | | | | This is a new class, so let's not support the old version strings and saddle ourselves with tech debt right from the get-go.
* Be explicit with tuples for %s formattingJeff Widman2018-11-1826-45/+45
| | | | Fix #1633
* Stop using broker-errors for client-side problemsJeff Widman2018-11-183-38/+44
| | | | | | | | | | | | | `UnsupportedVersionError` is intended to indicate a server-side error: https://github.com/dpkp/kafka-python/blob/ba7372e44ffa1ee49fb4d5efbd67534393e944db/kafka/errors.py#L375-L378 So we should not be raising it for client-side errors. I realize that semantically this seems like the appropriate error to raise. However, this is confusing when debugging... for a real-life example, see https://github.com/Parsely/pykafka/issues/697. So I strongly feel that server-side errors should be kept separate from client-side errors, even if all the client is doing is proactively protecting against hitting a situation where the broker would return this error.
* Update requirements-dev to latest versions used on travis runsDana Powers2018-11-171-11/+11
|
* Pin pytest on 3.x -- we arent ready for pytest 4.0 yetDana Powers2018-11-171-1/+1
|
* Use TypeError for invalid typeJeff Widman2018-11-171-1/+1
|
* Update changelogDana Powers2018-11-122-0/+106
|
* Migrate from `Unittest` to `pytest` (#1620)Jeff Widman2018-11-107-73/+34
|
* raising logging level on messages signalling data loss (#1553)Alexander Sibiryakov2018-11-101-2/+3
|
* set socket timeout for the wake_w (#1577)flaneur2018-11-102-0/+6
|
* (Attempt to) Fix deadlock between consumer and heartbeat (#1628)Dana Powers2018-11-102-4/+2
|
* Fix typoJeff Widman2018-11-071-1/+1
|
* Pre-compile pack/unpack function callsbillyevans2018-10-291-13/+29
| | | | | | | | | | | | | | | | I noticed that pack/unpack functions from https://github.com/dpkp/kafka-python/blob/master/kafka/protocol/types.py might be slightly improved. I made pre-compilation for them. It gives about 10% better performance compared to the current implementation. Consumption of 100msg: ``` 239884 0.187 0.000 0.287 0.000 types.py:18(_unpack) # new version 239884 0.192 0.000 0.323 0.000 types.py:17(_unpack) ``` I also made some profiling for producers/consumers. It gives about 1-1.5% time savings.
* Document KafkaAdmin classJeff Widman2018-10-293-0/+13
|
* Cleanup fixture importsJeff Widman2018-10-277-15/+10
| | | | | | | | | | | | | | | `random_string` now comes from `test.fixtures` and was being transparently imported via `test.testutil` so this bypasses the pointless indirect import. Similarly, `kafka_version` was transparently imported by `test.testutil` from `test.fixtures`. Also removed `random_port()` in `test.testutil` because its unused as its been replaced by the one in `test.fixtures`. This is part of the pytest migration that was started back in a1869c4be5f47b4f6433610249aaf29af4ec95e5.
* Minor aesthetic cleanup of partitioner testsJeff Widman2018-10-271-11/+10
|
* Remove unused ivy_root variableJeff Widman2018-10-271-1/+0
| | | | This is no longer used anywhere in the codebase
* Minor cleanup of testing docJeff Widman2018-10-271-13/+17
| | | | | | | Removed some of the hardcoded values as they are now outdated, and just pointed to where to find the current value in the code. Also some minor wordsmithing.
* Stop using deprecated log.warn()Jeff Widman2018-10-261-3/+3
| | | | I missed this in my previous cleanup back in 9221fcf83528b5c3657e43636cb84c1d18025acd.
View Lorry Controller Status