Update docs w/ KafkaProducer; move Simple clients to separate document

1 files changed, 41 insertions, 22 deletions

diff --git a/README.rst b/README.rst
index 2bcc150..61b737f 100644
--- a/README.rst
+++ b/README.rst
@@ -12,39 +12,58 @@ Kafka Python client
 .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg
     :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE
 
->>> pip install kafka-python
+Python client for the Apache Kafka distributed stream processing system.
+kafka-python is designed to function much like the official java client, with a
+sprinkling of pythonic interfaces (e.g., consumer iterators).
+
+kafka-python is best used with 0.9 brokers, but is backwards-compatible with
+older versions (to 0.8.0). Some features will only be enabled on newer brokers,
+however; for example, fully coordinated consumer groups -- i.e., dynamic partition
+assignment to multiple consumers in the same group -- requires use of 0.9 kafka
+brokers. Supporting this feature for earlier broker releases would require
+writing and maintaining custom leadership election and membership / health
+check code (perhaps using zookeeper or consul). For older brokers, you can
+achieve something similar by manually assigning different partitions to each
+consumer instance with config management tools like chef, ansible, etc. This
+approach will work fine, though it does not support rebalancing on failures.
+See `Compatibility <http://kafka-python.readthedocs.org/en/master/compatibility.html>`_
+for more details.
 
-kafka-python is a client for the Apache Kafka distributed stream processing
-system. It is designed to function much like the official java client, with a
-sprinkling of pythonic interfaces (e.g., iterators).
+Please note that the master branch may contain unreleased features. For release
+documentation, please see readthedocs and/or python's inline help.
 
+>>> pip install kafka-python
 
 KafkaConsumer
 *************
 
+KafkaConsumer is a high-level message consumer, intended to operate as similarly
+as possible to the official 0.9 java client. Full support for coordinated
+consumer groups requires use of kafka brokers that support the 0.9 Group APIs.
+
+See `ReadTheDocs <http://kafka-python.readthedocs.org/en/master/apidoc/KafkaConsumer.html>`_
+for API and configuration details.
+
+The consumer iterator returns ConsumerRecords, which are simple namedtuples
+that expose basic message attributes: topic, partition, offset, key, and value:
+
 >>> from kafka import KafkaConsumer
 >>> consumer = KafkaConsumer('my_favorite_topic')
 >>> for msg in consumer:
 ...     print (msg)
 
-KafkaConsumer is a full-featured,
-high-level message consumer class that is similar in design and function to the
-new 0.9 java consumer. Most configuration parameters defined by the official
-java client are supported as optional kwargs, with generally similar behavior.
-Gzip and Snappy compressed messages are supported transparently.
-
-In addition to the standard KafkaConsumer.poll() interface (which returns
-micro-batches of messages, grouped by topic-partition), kafka-python supports
-single-message iteration, yielding ConsumerRecord namedtuples, which include
-the topic, partition, offset, key, and value of each message.
-
-By default, KafkaConsumer will attempt to auto-commit
-message offsets every 5 seconds. When used with 0.9 kafka brokers,
-KafkaConsumer will dynamically assign partitions using
-the kafka GroupCoordinator APIs and a RoundRobinPartitionAssignor
-partitioning strategy, enabling relatively straightforward parallel consumption
-patterns. See `ReadTheDocs <http://kafka-python.readthedocs.org/master/>`_
-for examples.
+>>> # manually assign the partition list for the consumer
+>>> from kafka import TopicPartition
+>>> consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
+>>> consumer.assign([TopicPartition('foobar', 2)])
+>>> msg = next(consumer)
+
+>>> # Deserialize msgpack-encoded values
+>>> consumer = KafkaConsumer(value_deserializer=msgpack.dumps)
+>>> consumer.subscribe(['msgpackfoo'])
+>>> for msg in consumer:
+...     msg = next(consumer)
+...     assert isinstance(msg.value, dict)
 
 
 KafkaProducer