From b0c59303fc71c287b001c83ca123d57c58085f1a Mon Sep 17 00:00:00 2001 From: Jeff Widman Date: Sun, 15 Nov 2020 12:10:12 -0800 Subject: Cleanup install instructions for optional libs (#2139) This cleans up the install instructions to specify that optional libs should be installed using the `kafka-python[opt]` format. This leverages `setuptools`' `extra_requires` feature to let our users choose to inherit any versions pins we might apply to our optional dependencies. This also re-orders the docs slightly to give more visibility to the `crc32c` install because users are unlikely to realize it exists. It also cleans up the formatting of the compression libraries slightly, as they were getting a little unwieldy. --- README.rst | 23 +++++++++++++++++------ docs/index.rst | 25 ++++++++++++++++++++----- docs/install.rst | 40 ++++++++++++++++++++-------------------- 3 files changed, 57 insertions(+), 31 deletions(-) diff --git a/README.rst b/README.rst index 812e152..5f83444 100644 --- a/README.rst +++ b/README.rst @@ -34,6 +34,7 @@ documentation, please see readthedocs and/or python's inline help. >>> pip install kafka-python + KafkaConsumer ************* @@ -78,6 +79,7 @@ that expose basic message attributes: topic, partition, offset, key, and value: >>> # Get consumer metrics >>> metrics = consumer.metrics() + KafkaProducer ************* @@ -124,6 +126,7 @@ for more details. >>> # Get producer performance metrics >>> metrics = producer.metrics() + Thread safety ************* @@ -133,14 +136,20 @@ KafkaConsumer which cannot. While it is possible to use the KafkaConsumer in a thread-local manner, multiprocessing is recommended. + Compression *********** -kafka-python supports gzip compression/decompression natively. To produce or consume lz4 -compressed messages, you should install python-lz4 (pip install lz4). -To enable snappy compression/decompression install python-snappy (also requires snappy library). -See -for more information. +kafka-python supports the following compression formats: + +- gzip +- LZ4 +- Snappy +- Zstandard (zstd) + +gzip is supported natively, the others require installing additional libraries. +See for more information. + Optimized CRC32 Validation ************************** @@ -148,7 +157,9 @@ Optimized CRC32 Validation Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure python implementation for compatibility. To improve performance for high-throughput applications, kafka-python will use `crc32c` for optimized native code if installed. -See https://pypi.org/project/crc32c/ +See for installation instructions. +See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib. + Protocol ******** diff --git a/docs/index.rst b/docs/index.rst index 536a058..1f2a4ce 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -33,6 +33,7 @@ documentation, please see readthedocs and/or python's inline help. >>> pip install kafka-python + KafkaConsumer ************* @@ -122,12 +123,26 @@ multiprocessing is recommended. Compression *********** -kafka-python supports multiple compression types: +kafka-python supports the following compression formats: + + - gzip + - LZ4 + - Snappy + - Zstandard (zstd) + +gzip is supported natively, the others require installing additional libraries. +See `Install `_ for more information. + + +Optimized CRC32 Validation +************************** + +Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure +python implementation for compatibility. To improve performance for high-throughput +applications, kafka-python will use `crc32c` for optimized native code if installed. +See `Install `_ for installation instructions and +https://pypi.org/project/crc32c/ for details on the underlying crc32c lib. - - gzip : supported natively - - lz4 : requires `python-lz4 `_ installed - - snappy : requires the `python-snappy `_ package (which requires the snappy C library) - - zstd : requires the `python-zstandard `_ package installed Protocol ******** diff --git a/docs/install.rst b/docs/install.rst index 200ca17..19901ee 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -23,20 +23,33 @@ Bleeding-Edge pip install ./kafka-python -Optional LZ4 install +Optional crc32c install +*********************** +Highly recommended if you are using Kafka 11+ brokers. For those `kafka-python` +uses a new message protocol version, that requires calculation of `crc32c`, +which differs from the `zlib.crc32` hash implementation. By default `kafka-python` +calculates it in pure python, which is quite slow. To speed it up we optionally +support https://pypi.python.org/pypi/crc32c package if it's installed. + +.. code:: bash + + pip install 'kafka-python[crc32c]' + + +Optional ZSTD install ******************** -To enable LZ4 compression/decompression, install python-lz4: +To enable ZSTD compression/decompression, install python-zstandard: ->>> pip install lz4 +>>> pip install 'kafka-python[zstd]' -Optional crc32c install +Optional LZ4 install ******************** -To enable optimized CRC32 checksum validation, install crc32c: +To enable LZ4 compression/decompression, install python-lz4: ->>> pip install crc32c +>>> pip install 'kafka-python[lz4]' Optional Snappy install @@ -77,17 +90,4 @@ Install the `python-snappy` module .. code:: bash - pip install python-snappy - - -Optional crc32c install -*********************** -Highly recommended if you are using Kafka 11+ brokers. For those `kafka-python` -uses a new message protocol version, that requires calculation of `crc32c`, -which differs from `zlib.crc32` hash implementation. By default `kafka-python` -calculates it in pure python, which is quite slow. To speed it up we optionally -support https://pypi.python.org/pypi/crc32c package if it's installed. - -.. code:: bash - - pip install crc32c + pip install 'kafka-python[snappy]' -- cgit v1.2.1