From 2ca15fedbec25233240d0cff267c14127d5f6c81 Mon Sep 17 00:00:00 2001 From: Akihiro Motoki Date: Fri, 7 Jul 2017 00:27:45 +0000 Subject: rearrange existing documentation to fit the new standard layout Change-Id: If03b33fc1393c3a04d7d155b31e8377dc9e09360 --- doc/source/compatibility.rst | 68 -------- doc/source/developers.rst | 82 --------- doc/source/drivers.rst | 233 -------------------------- doc/source/history.rst | 1 - doc/source/index.rst | 24 +-- doc/source/install.rst | 43 ----- doc/source/install/index.rst | 43 +++++ doc/source/reference/index.rst | 82 +++++++++ doc/source/tutorial/coordinator.rst | 42 ----- doc/source/tutorial/group_membership.rst | 41 ----- doc/source/tutorial/hashring.rst | 10 -- doc/source/tutorial/index.rst | 16 -- doc/source/tutorial/leader_election.rst | 25 --- doc/source/tutorial/lock.rst | 14 -- doc/source/tutorial/partitioner.rst | 11 -- doc/source/user/compatibility.rst | 68 ++++++++ doc/source/user/drivers.rst | 233 ++++++++++++++++++++++++++ doc/source/user/history.rst | 1 + doc/source/user/index.rst | 15 ++ doc/source/user/tutorial/coordinator.rst | 42 +++++ doc/source/user/tutorial/group_membership.rst | 41 +++++ doc/source/user/tutorial/hashring.rst | 10 ++ doc/source/user/tutorial/index.rst | 16 ++ doc/source/user/tutorial/leader_election.rst | 25 +++ doc/source/user/tutorial/lock.rst | 14 ++ doc/source/user/tutorial/partitioner.rst | 11 ++ tox.ini | 2 +- 27 files changed, 607 insertions(+), 606 deletions(-) delete mode 100644 doc/source/compatibility.rst delete mode 100644 doc/source/developers.rst delete mode 100644 doc/source/drivers.rst delete mode 100644 doc/source/history.rst delete mode 100644 doc/source/install.rst create mode 100644 doc/source/install/index.rst create mode 100644 doc/source/reference/index.rst delete mode 100644 doc/source/tutorial/coordinator.rst delete mode 100644 doc/source/tutorial/group_membership.rst delete mode 100644 doc/source/tutorial/hashring.rst delete mode 100644 doc/source/tutorial/index.rst delete mode 100644 doc/source/tutorial/leader_election.rst delete mode 100644 doc/source/tutorial/lock.rst delete mode 100644 doc/source/tutorial/partitioner.rst create mode 100644 doc/source/user/compatibility.rst create mode 100644 doc/source/user/drivers.rst create mode 100644 doc/source/user/history.rst create mode 100644 doc/source/user/index.rst create mode 100644 doc/source/user/tutorial/coordinator.rst create mode 100644 doc/source/user/tutorial/group_membership.rst create mode 100644 doc/source/user/tutorial/hashring.rst create mode 100644 doc/source/user/tutorial/index.rst create mode 100644 doc/source/user/tutorial/leader_election.rst create mode 100644 doc/source/user/tutorial/lock.rst create mode 100644 doc/source/user/tutorial/partitioner.rst diff --git a/doc/source/compatibility.rst b/doc/source/compatibility.rst deleted file mode 100644 index 975bedb..0000000 --- a/doc/source/compatibility.rst +++ /dev/null @@ -1,68 +0,0 @@ -============= -Compatibility -============= - -Grouping -======== - -APIs ----- - -* :py:meth:`~tooz.coordination.CoordinationDriver.watch_join_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.unwatch_join_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.watch_leave_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.unwatch_leave_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.create_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.get_groups` -* :py:meth:`~tooz.coordination.CoordinationDriver.join_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.leave_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.delete_group` -* :py:meth:`~tooz.coordination.CoordinationDriver.get_members` -* :py:meth:`~tooz.coordination.CoordinationDriver.get_member_capabilities` -* :py:meth:`~tooz.coordination.CoordinationDriver.update_capabilities` - -Driver support --------------- - -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= -:py:class:`~tooz.drivers.consul.ConsulDriver` :py:class:`~tooz.drivers.etcd.EtcdDriver` :py:class:`~tooz.drivers.file.FileDriver` :py:class:`~tooz.drivers.ipc.IPCDriver` :py:class:`~tooz.drivers.memcached.MemcachedDriver` :py:class:`~tooz.drivers.mysql.MySQLDriver` :py:class:`~tooz.drivers.pgsql.PostgresDriver` :py:class:`~tooz.drivers.redis.RedisDriver` :py:class:`~tooz.drivers.zake.ZakeDriver` :py:class:`~tooz.drivers.zookeeper.KazooDriver` -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= -No No Yes No Yes No No Yes Yes Yes -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= - -Leaders -======= - -APIs ----- - -* :py:meth:`~tooz.coordination.CoordinationDriver.watch_elected_as_leader` -* :py:meth:`~tooz.coordination.CoordinationDriver.unwatch_elected_as_leader` -* :py:meth:`~tooz.coordination.CoordinationDriver.stand_down_group_leader` -* :py:meth:`~tooz.coordination.CoordinationDriver.get_leader` - -Driver support --------------- - -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= -:py:class:`~tooz.drivers.consul.ConsulDriver` :py:class:`~tooz.drivers.etcd.EtcdDriver` :py:class:`~tooz.drivers.file.FileDriver` :py:class:`~tooz.drivers.ipc.IPCDriver` :py:class:`~tooz.drivers.memcached.MemcachedDriver` :py:class:`~tooz.drivers.mysql.MySQLDriver` :py:class:`~tooz.drivers.pgsql.PostgresDriver` :py:class:`~tooz.drivers.redis.RedisDriver` :py:class:`~tooz.drivers.zake.ZakeDriver` :py:class:`~tooz.drivers.zookeeper.KazooDriver` -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= -No No No No Yes No No Yes Yes Yes -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= - -Locking -======= - -APIs ----- - -* :py:meth:`~tooz.coordination.CoordinationDriver.get_lock` - -Driver support --------------- - -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= -:py:class:`~tooz.drivers.consul.ConsulDriver` :py:class:`~tooz.drivers.etcd.EtcdDriver` :py:class:`~tooz.drivers.file.FileDriver` :py:class:`~tooz.drivers.ipc.IPCDriver` :py:class:`~tooz.drivers.memcached.MemcachedDriver` :py:class:`~tooz.drivers.mysql.MySQLDriver` :py:class:`~tooz.drivers.pgsql.PostgresDriver` :py:class:`~tooz.drivers.redis.RedisDriver` :py:class:`~tooz.drivers.zake.ZakeDriver` :py:class:`~tooz.drivers.zookeeper.KazooDriver` -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= -Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes -=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= diff --git a/doc/source/developers.rst b/doc/source/developers.rst deleted file mode 100644 index 57ccbec..0000000 --- a/doc/source/developers.rst +++ /dev/null @@ -1,82 +0,0 @@ -========== -Developers -========== - -Interfaces ----------- - -.. autoclass:: tooz.coordination.CoordinationDriver - :members: - -Consul -~~~~~~ - -.. autoclass:: tooz.drivers.consul.ConsulDriver - :members: - -Etcd -~~~~ - -.. autoclass:: tooz.drivers.etcd.EtcdDriver - :members: - -File -~~~~ - -.. autoclass:: tooz.drivers.file.FileDriver - :members: - -IPC -~~~ - -.. autoclass:: tooz.drivers.ipc.IPCDriver - :members: - -Memcached -~~~~~~~~~ - -.. autoclass:: tooz.drivers.memcached.MemcachedDriver - :members: - -Mysql -~~~~~ - -.. autoclass:: tooz.drivers.mysql.MySQLDriver - :members: - -PostgreSQL -~~~~~~~~~~ - -.. autoclass:: tooz.drivers.pgsql.PostgresDriver - :members: - -Redis -~~~~~ - -.. autoclass:: tooz.drivers.redis.RedisDriver - :members: - -Zake -~~~~ - -.. autoclass:: tooz.drivers.zake.ZakeDriver - :members: - -Zookeeper -~~~~~~~~~ - -.. autoclass:: tooz.drivers.zookeeper.KazooDriver - :members: - -Exceptions ----------- - -.. autoclass:: tooz.ToozError -.. autoclass:: tooz.coordination.ToozConnectionError -.. autoclass:: tooz.coordination.OperationTimedOut -.. autoclass:: tooz.coordination.GroupNotCreated -.. autoclass:: tooz.coordination.GroupAlreadyExist -.. autoclass:: tooz.coordination.MemberAlreadyExist -.. autoclass:: tooz.coordination.MemberNotJoined -.. autoclass:: tooz.coordination.GroupNotEmpty -.. autofunction:: tooz.utils.raise_with_cause diff --git a/doc/source/drivers.rst b/doc/source/drivers.rst deleted file mode 100644 index 3d3fdd3..0000000 --- a/doc/source/drivers.rst +++ /dev/null @@ -1,233 +0,0 @@ -======= -Drivers -======= - -Tooz is provided with several drivers implementing the provided coordination -API. While all drivers provides the same set of features with respect to the -API, some of them have different characteristics: - -Zookeeper ---------- - -**Driver:** :py:class:`tooz.drivers.zookeeper.KazooDriver` - -**Characteristics:** - -:py:attr:`tooz.drivers.zookeeper.KazooDriver.CHARACTERISTICS` - -**Entrypoint name:** ``zookeeper`` or ``kazoo`` - -**Summary:** - -The zookeeper is the reference implementation and provides the most solid -features as it's possible to build a cluster of zookeeper servers that is -resilient towards network partitions for example. - -**Test driver:** :py:class:`tooz.drivers.zake.ZakeDriver` - -**Characteristics:** - -:py:attr:`tooz.drivers.zake.ZakeDriver.CHARACTERISTICS` - -**Test driver entrypoint name:** ``zake`` - -Considerations -~~~~~~~~~~~~~~ - -- Primitives are based on sessions (and typically require careful selection - of session heartbeat periodicity and server side configuration of session - expiry). - -Memcached ---------- - -**Driver:** :py:class:`tooz.drivers.memcached.MemcachedDriver` - -**Characteristics:** - -:py:attr:`tooz.drivers.memcached.MemcachedDriver.CHARACTERISTICS` - -**Entrypoint name:** ``memcached`` - -**Summary:** - -The memcached driver is a basic implementation and provides little -resiliency, though it's much simpler to setup. A lot of the features provided -in tooz are based on timeout (heartbeats, locks, etc) so are less resilient -than other backends. - -Considerations -~~~~~~~~~~~~~~ - -- Less resilient than other backends such as zookeeper and redis. -- Primitives are often based on TTL(s) that may expire before - being renewed. -- Lacks certain primitives (compare and delete) so certain functionality - is fragile and/or broken due to this. - -Redis ------ - -**Driver:** :py:class:`tooz.drivers.redis.RedisDriver` - -**Characteristics:** - -:py:attr:`tooz.drivers.redis.RedisDriver.CHARACTERISTICS` - -**Entrypoint name:** ``redis`` - -**Summary:** - -The redis driver is a basic implementation and provides reasonable resiliency -when used with `redis-sentinel`_. A lot of the features provided in tooz are -based on timeout (heartbeats, locks, etc) so are less resilient than other -backends. - -Considerations -~~~~~~~~~~~~~~ - -- Less resilient than other backends such as zookeeper. -- Primitives are often based on TTL(s) that may expire before - being renewed. - -IPC ---- - -**Driver:** :py:class:`tooz.drivers.ipc.IPCDriver` - -**Characteristics:** :py:attr:`tooz.drivers.ipc.IPCDriver.CHARACTERISTICS` - -**Entrypoint name:** ``ipc`` - -**Summary:** - -The IPC driver is based on Posix IPC API and implements a lock -mechanism and some basic group primitives (with **huge** -limitations). - -Considerations -~~~~~~~~~~~~~~ - -- The lock can **only** be distributed locally to a computer - processes. - -File ----- - -**Driver:** :py:class:`tooz.drivers.file.FileDriver` - -**Characteristics:** :py:attr:`tooz.drivers.file.FileDriver.CHARACTERISTICS` - -**Entrypoint name:** ``file`` - -**Summary:** - -The file driver is a **simple** driver based on files and directories. It -implements a lock based on POSIX or Window file level locking -mechanism and some basic group primitives (with **huge** -limitations). - -Considerations -~~~~~~~~~~~~~~ - -- The lock can **only** be distributed locally to a computer processes. -- Certain concepts provided by it are **not** crash tolerant. - -PostgreSQL ----------- - -**Driver:** :py:class:`tooz.drivers.pgsql.PostgresDriver` - -**Characteristics:** - -:py:attr:`tooz.drivers.pgsql.PostgresDriver.CHARACTERISTICS` - -**Entrypoint name:** ``postgresql`` - -**Summary:** - -The postgresql driver is a driver providing only a distributed lock (for now) -and is based on the `PostgreSQL database server`_ and its API(s) that provide -for `advisory locks`_ to be created and used by applications. When a lock is -acquired it will release either when explicitly released or automatically when -the database session ends (for example if the program using the lock crashes). - -Considerations -~~~~~~~~~~~~~~ - -- Lock that may be acquired restricted by - ``max_locks_per_transaction * (max_connections + max_prepared_transactions)`` - upper bound (PostgreSQL server configuration settings). - -MySQL ------ - -**Driver:** :py:class:`tooz.drivers.mysql.MySQLDriver` - -**Characteristics:** :py:attr:`tooz.drivers.mysql.MySQLDriver.CHARACTERISTICS` - -**Entrypoint name:** ``mysql`` - -**Summary:** - -The MySQL driver is a driver providing only distributed locks (for now) -and is based on the `MySQL database server`_ supported `get_lock`_ -primitives. When a lock is acquired it will release either when explicitly -released or automatically when the database session ends (for example if -the program using the lock crashes). - -Considerations -~~~~~~~~~~~~~~ - -- Does **not** work correctly on some MySQL versions. -- Does **not** work when MySQL replicates from one server to another (locks - are local to the server that they were created from). - -Etcd ----- - -**Driver:** :py:class:`tooz.drivers.etcd.EtcdDriver` - -**Characteristics:** :py:attr:`tooz.drivers.etcd.EtcdDriver.CHARACTERISTICS` - -**Entrypoint name:** ``etcd`` - -**Summary:** - -The etcd driver is a driver providing only distributed locks (for now) -and is based on the `etcd server`_ supported key/value storage and -associated primitives. - -Consul ------- - -**Driver:** :py:class:`tooz.drivers.consul.ConsulDriver` - -**Characteristics:** - -:py:attr:`tooz.drivers.consul.ConsulDriver.CHARACTERISTICS` - -**Entrypoint name:** ``consul`` - -**Summary:** - -The `consul`_ driver is a driver providing only distributed locks (for now) -and is based on the consul server key/value storage and/or -primitives. When a lock is acquired it will release either when explicitly -released or automatically when the consul session ends (for example if -the program using the lock crashes). - -Characteristics ---------------- - -.. autoclass:: tooz.coordination.Characteristics - -.. _etcd server: https://coreos.com/etcd/ -.. _consul: https://www.consul.io/ -.. _advisory locks: http://www.postgresql.org/docs/8.2/interactive/\ - explicit-locking.html#ADVISORY-LOCKS -.. _get_lock: http://dev.mysql.com/doc/refman/5.5/en/\ - miscellaneous-functions.html#function_get-lock -.. _PostgreSQL database server: http://postgresql.org -.. _MySQL database server: http://mysql.org -.. _redis-sentinel: http://redis.io/topics/sentinel diff --git a/doc/source/history.rst b/doc/source/history.rst deleted file mode 100644 index 69ed4fe..0000000 --- a/doc/source/history.rst +++ /dev/null @@ -1 +0,0 @@ -.. include:: ../../ChangeLog diff --git a/doc/source/index.rst b/doc/source/index.rst index 8bb1c84..af18503 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -6,28 +6,14 @@ The Tooz project aims at centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications. [#f1]_ -Contents -======== - -.. toctree:: - :glob: - - install - drivers - compatibility - tutorial/index - developers - -Release Notes -============= - .. toctree:: - :maxdepth: 1 + :maxdepth: 2 - history + install/index + user/index + reference/index -Indices and tables -================== +.. rubric:: Indices and tables * :ref:`genindex` * :ref:`modindex` diff --git a/doc/source/install.rst b/doc/source/install.rst deleted file mode 100644 index 3070b4a..0000000 --- a/doc/source/install.rst +++ /dev/null @@ -1,43 +0,0 @@ -============ -Installation -============ - -Python Versions -=============== - -Tooz is tested under Python 2.7 and 3.4. - -.. _install-basic: - -Basic Installation -================== - -Tooz should be installed into the same site-packages area where -the application and extensions are installed (either a virtualenv or -the global site-packages). You may need administrative privileges to -do that. The easiest way to install it is using pip_:: - - $ pip install tooz - -or:: - - $ sudo pip install tooz - -.. _pip: http://pypi.python.org/pypi/pip - -Download -======== - -Tooz releases are hosted on PyPI and can be downloaded from: -http://pypi.python.org/pypi/tooz - -Source Code -=========== - -The source is hosted on the OpenStack infrastructure: https://git.openstack.org/cgit/openstack/tooz/ - -Reporting Bugs -============== - -Please report bugs through the launchpad project: -https://launchpad.net/python-tooz diff --git a/doc/source/install/index.rst b/doc/source/install/index.rst new file mode 100644 index 0000000..3070b4a --- /dev/null +++ b/doc/source/install/index.rst @@ -0,0 +1,43 @@ +============ +Installation +============ + +Python Versions +=============== + +Tooz is tested under Python 2.7 and 3.4. + +.. _install-basic: + +Basic Installation +================== + +Tooz should be installed into the same site-packages area where +the application and extensions are installed (either a virtualenv or +the global site-packages). You may need administrative privileges to +do that. The easiest way to install it is using pip_:: + + $ pip install tooz + +or:: + + $ sudo pip install tooz + +.. _pip: http://pypi.python.org/pypi/pip + +Download +======== + +Tooz releases are hosted on PyPI and can be downloaded from: +http://pypi.python.org/pypi/tooz + +Source Code +=========== + +The source is hosted on the OpenStack infrastructure: https://git.openstack.org/cgit/openstack/tooz/ + +Reporting Bugs +============== + +Please report bugs through the launchpad project: +https://launchpad.net/python-tooz diff --git a/doc/source/reference/index.rst b/doc/source/reference/index.rst new file mode 100644 index 0000000..15b7d57 --- /dev/null +++ b/doc/source/reference/index.rst @@ -0,0 +1,82 @@ +================ +Module Reference +================ + +Interfaces +---------- + +.. autoclass:: tooz.coordination.CoordinationDriver + :members: + +Consul +~~~~~~ + +.. autoclass:: tooz.drivers.consul.ConsulDriver + :members: + +Etcd +~~~~ + +.. autoclass:: tooz.drivers.etcd.EtcdDriver + :members: + +File +~~~~ + +.. autoclass:: tooz.drivers.file.FileDriver + :members: + +IPC +~~~ + +.. autoclass:: tooz.drivers.ipc.IPCDriver + :members: + +Memcached +~~~~~~~~~ + +.. autoclass:: tooz.drivers.memcached.MemcachedDriver + :members: + +Mysql +~~~~~ + +.. autoclass:: tooz.drivers.mysql.MySQLDriver + :members: + +PostgreSQL +~~~~~~~~~~ + +.. autoclass:: tooz.drivers.pgsql.PostgresDriver + :members: + +Redis +~~~~~ + +.. autoclass:: tooz.drivers.redis.RedisDriver + :members: + +Zake +~~~~ + +.. autoclass:: tooz.drivers.zake.ZakeDriver + :members: + +Zookeeper +~~~~~~~~~ + +.. autoclass:: tooz.drivers.zookeeper.KazooDriver + :members: + +Exceptions +---------- + +.. autoclass:: tooz.ToozError +.. autoclass:: tooz.coordination.ToozConnectionError +.. autoclass:: tooz.coordination.OperationTimedOut +.. autoclass:: tooz.coordination.GroupNotCreated +.. autoclass:: tooz.coordination.GroupAlreadyExist +.. autoclass:: tooz.coordination.MemberAlreadyExist +.. autoclass:: tooz.coordination.MemberNotJoined +.. autoclass:: tooz.coordination.GroupNotEmpty +.. autofunction:: tooz.utils.raise_with_cause diff --git a/doc/source/tutorial/coordinator.rst b/doc/source/tutorial/coordinator.rst deleted file mode 100644 index faf9ab2..0000000 --- a/doc/source/tutorial/coordinator.rst +++ /dev/null @@ -1,42 +0,0 @@ -======================= - Creating A Coordinator -======================= - -The principal object provided by tooz is the *coordinator*. It allows you to -use various features, such as group membership, leader election or -distributed locking. - -The features provided by tooz coordinator are implemented using different -drivers. When creating a coordinator, you need to specify which back-end -driver you want it to use. Different drivers may provide different set of -capabilities. - -If a driver does not support a feature, it will raise a -:class:`~tooz.NotImplemented` exception. - -This example program loads a basic coordinator using the ZooKeeper based -driver. - -.. literalinclude:: ../../../examples/coordinator.py - :language: python - -The second argument passed to the coordinator must be a unique identifier -identifying the running program. - -After the coordinator is created, it can be used to use the various features -provided. - -In order to keep the connection to the coordination server active, the method -:meth:`~tooz.coordination.CoordinationDriver.heartbeat` method must be called -regularly. This will ensure that the coordinator is not considered dead by -other program participating in the coordination. Unless you want to call it -manually, you can use tooz builtin heartbeat manager by passing the -`start_heart` argument. - -.. literalinclude:: ../../../examples/coordinator_heartbeat.py - :language: python - -heartbeat at different moment or intervals. - -Note that certain drivers, such as `memcached` are heavily based on timeout, -so the interval used to run the heartbeat is important. diff --git a/doc/source/tutorial/group_membership.rst b/doc/source/tutorial/group_membership.rst deleted file mode 100644 index d16da8c..0000000 --- a/doc/source/tutorial/group_membership.rst +++ /dev/null @@ -1,41 +0,0 @@ -===================== - Group Membership -===================== - -Basic operations -=================== - -One of the feature provided by the coordinator is the ability to handle -group membership. Once a group is created, any coordinator can join the -group and become a member of it. Any coordinator can be notified when a -members joins or leaves the group. - -.. literalinclude:: ../../../examples/group_membership.py - :language: python - -Note that all the operation are asynchronous. That means you cannot be sure -that your group has been created or joined before you call the -:meth:`tooz.coordination.CoordAsyncResult.get` method. - -You can also leave a group using the -:meth:`tooz.coordination.CoordinationDriver.leave_group` method. The list of -all available groups is retrievable via the -:meth:`tooz.coordination.CoordinationDriver.get_groups` method. - -Watching Group Changes -====================== -It's possible to watch and get notified when the member list of a group -changes. That's useful to run callback functions whenever something happens -in that group. - - -.. literalinclude:: ../../../examples/group_membership_watch.py - :language: python - -Using :meth:`tooz.coordination.CoordinationDriver.watch_join_group` and -:meth:`tooz.coordination.CoordinationDriver.watch_leave_group` your -application can be notified each time a member join or leave a group. To -stop watching an event, the two methods -:meth:`tooz.coordination.CoordinationDriver.unwatch_join_group` and -:meth:`tooz.coordination.CoordinationDriver.unwatch_leave_group` allow to -unregister a particular callback. diff --git a/doc/source/tutorial/hashring.rst b/doc/source/tutorial/hashring.rst deleted file mode 100644 index 0adc573..0000000 --- a/doc/source/tutorial/hashring.rst +++ /dev/null @@ -1,10 +0,0 @@ -=========== - Hash ring -=========== - -Tooz provides a consistent hash ring implementation. It can be used to map -objects (represented via binary keys) to one or several nodes. When the node -list changes, the rebalancing of objects across the ring is kept minimal. - -.. literalinclude:: ../../../examples/hashring.py - :language: python diff --git a/doc/source/tutorial/index.rst b/doc/source/tutorial/index.rst deleted file mode 100644 index c2d398c..0000000 --- a/doc/source/tutorial/index.rst +++ /dev/null @@ -1,16 +0,0 @@ -===================================== - Using Tooz in Your Application -===================================== - -This tutorial is a step-by-step walk-through demonstrating how to -use tooz in your application. - -.. toctree:: - :maxdepth: 2 - - coordinator - group_membership - leader_election - lock - hashring - partitioner diff --git a/doc/source/tutorial/leader_election.rst b/doc/source/tutorial/leader_election.rst deleted file mode 100644 index 052e6c3..0000000 --- a/doc/source/tutorial/leader_election.rst +++ /dev/null @@ -1,25 +0,0 @@ -================= - Leader Election -================= - -Each group can elect its own leader. There can be only one leader at a time -in a group. Only members that are running for the election can be elected. -As soon as one of leader steps down or dies, a new member that was running -for the election will be elected. - -.. literalinclude:: ../../../examples/leader_election.py - :language: python - -The method -:meth:`tooz.coordination.CoordinationDriver.watch_elected_as_leader` allows -to register for a function to be called back when the member is elected as a -leader. Using this function indicates that the run is therefore running for -the election. The member can stop running by unregistering all its callbacks -with :meth:`tooz.coordination.CoordinationDriver.unwatch_elected_as_leader`. -It can also temporarily try to step down as a leader with -:meth:`tooz.coordination.CoordinationDriver.stand_down_group_leader`. If -another member is in the run for election, it may be elected instead. - -To retrieve the leader of a group, even when not being part of the group, -the method :meth:`tooz.coordination.CoordinationDriver.get_leader()` can be -used. diff --git a/doc/source/tutorial/lock.rst b/doc/source/tutorial/lock.rst deleted file mode 100644 index cf34d52..0000000 --- a/doc/source/tutorial/lock.rst +++ /dev/null @@ -1,14 +0,0 @@ -====== - Lock -====== - -Tooz provides distributed locks. A lock is identified by a name, and a lock can -only be acquired by one coordinator at a time. - -.. literalinclude:: ../../../examples/lock.py - :language: python - -The method :meth:`tooz.coordination.CoordinationDriver.get_lock` allows -to create a lock identified by a name. Once you retrieve this lock, you can -use it as a context manager or use the :meth:`tooz.locking.Lock.acquire` and -:meth:`tooz.locking.Lock.release` methods to acquire and release the lock. diff --git a/doc/source/tutorial/partitioner.rst b/doc/source/tutorial/partitioner.rst deleted file mode 100644 index d8547b8..0000000 --- a/doc/source/tutorial/partitioner.rst +++ /dev/null @@ -1,11 +0,0 @@ -============= - Partitioner -============= - -Tooz provides a partitioner object based on its consistent hash ring -implementation. It can be used to map Python objects to one or several nodes. -The partitioner object automatically keeps track of nodes joining and leaving -the group, so the rebalancing is managed. - -.. literalinclude:: ../../../examples/partitioner.py - :language: python diff --git a/doc/source/user/compatibility.rst b/doc/source/user/compatibility.rst new file mode 100644 index 0000000..975bedb --- /dev/null +++ b/doc/source/user/compatibility.rst @@ -0,0 +1,68 @@ +============= +Compatibility +============= + +Grouping +======== + +APIs +---- + +* :py:meth:`~tooz.coordination.CoordinationDriver.watch_join_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.unwatch_join_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.watch_leave_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.unwatch_leave_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.create_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.get_groups` +* :py:meth:`~tooz.coordination.CoordinationDriver.join_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.leave_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.delete_group` +* :py:meth:`~tooz.coordination.CoordinationDriver.get_members` +* :py:meth:`~tooz.coordination.CoordinationDriver.get_member_capabilities` +* :py:meth:`~tooz.coordination.CoordinationDriver.update_capabilities` + +Driver support +-------------- + +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= +:py:class:`~tooz.drivers.consul.ConsulDriver` :py:class:`~tooz.drivers.etcd.EtcdDriver` :py:class:`~tooz.drivers.file.FileDriver` :py:class:`~tooz.drivers.ipc.IPCDriver` :py:class:`~tooz.drivers.memcached.MemcachedDriver` :py:class:`~tooz.drivers.mysql.MySQLDriver` :py:class:`~tooz.drivers.pgsql.PostgresDriver` :py:class:`~tooz.drivers.redis.RedisDriver` :py:class:`~tooz.drivers.zake.ZakeDriver` :py:class:`~tooz.drivers.zookeeper.KazooDriver` +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= +No No Yes No Yes No No Yes Yes Yes +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= + +Leaders +======= + +APIs +---- + +* :py:meth:`~tooz.coordination.CoordinationDriver.watch_elected_as_leader` +* :py:meth:`~tooz.coordination.CoordinationDriver.unwatch_elected_as_leader` +* :py:meth:`~tooz.coordination.CoordinationDriver.stand_down_group_leader` +* :py:meth:`~tooz.coordination.CoordinationDriver.get_leader` + +Driver support +-------------- + +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= +:py:class:`~tooz.drivers.consul.ConsulDriver` :py:class:`~tooz.drivers.etcd.EtcdDriver` :py:class:`~tooz.drivers.file.FileDriver` :py:class:`~tooz.drivers.ipc.IPCDriver` :py:class:`~tooz.drivers.memcached.MemcachedDriver` :py:class:`~tooz.drivers.mysql.MySQLDriver` :py:class:`~tooz.drivers.pgsql.PostgresDriver` :py:class:`~tooz.drivers.redis.RedisDriver` :py:class:`~tooz.drivers.zake.ZakeDriver` :py:class:`~tooz.drivers.zookeeper.KazooDriver` +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= +No No No No Yes No No Yes Yes Yes +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= + +Locking +======= + +APIs +---- + +* :py:meth:`~tooz.coordination.CoordinationDriver.get_lock` + +Driver support +-------------- + +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= +:py:class:`~tooz.drivers.consul.ConsulDriver` :py:class:`~tooz.drivers.etcd.EtcdDriver` :py:class:`~tooz.drivers.file.FileDriver` :py:class:`~tooz.drivers.ipc.IPCDriver` :py:class:`~tooz.drivers.memcached.MemcachedDriver` :py:class:`~tooz.drivers.mysql.MySQLDriver` :py:class:`~tooz.drivers.pgsql.PostgresDriver` :py:class:`~tooz.drivers.redis.RedisDriver` :py:class:`~tooz.drivers.zake.ZakeDriver` :py:class:`~tooz.drivers.zookeeper.KazooDriver` +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= +Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes +=============================================== =========================================== =========================================== ========================================= ===================================================== ============================================= ================================================ ============================================= =========================================== ================================================= diff --git a/doc/source/user/drivers.rst b/doc/source/user/drivers.rst new file mode 100644 index 0000000..3d3fdd3 --- /dev/null +++ b/doc/source/user/drivers.rst @@ -0,0 +1,233 @@ +======= +Drivers +======= + +Tooz is provided with several drivers implementing the provided coordination +API. While all drivers provides the same set of features with respect to the +API, some of them have different characteristics: + +Zookeeper +--------- + +**Driver:** :py:class:`tooz.drivers.zookeeper.KazooDriver` + +**Characteristics:** + +:py:attr:`tooz.drivers.zookeeper.KazooDriver.CHARACTERISTICS` + +**Entrypoint name:** ``zookeeper`` or ``kazoo`` + +**Summary:** + +The zookeeper is the reference implementation and provides the most solid +features as it's possible to build a cluster of zookeeper servers that is +resilient towards network partitions for example. + +**Test driver:** :py:class:`tooz.drivers.zake.ZakeDriver` + +**Characteristics:** + +:py:attr:`tooz.drivers.zake.ZakeDriver.CHARACTERISTICS` + +**Test driver entrypoint name:** ``zake`` + +Considerations +~~~~~~~~~~~~~~ + +- Primitives are based on sessions (and typically require careful selection + of session heartbeat periodicity and server side configuration of session + expiry). + +Memcached +--------- + +**Driver:** :py:class:`tooz.drivers.memcached.MemcachedDriver` + +**Characteristics:** + +:py:attr:`tooz.drivers.memcached.MemcachedDriver.CHARACTERISTICS` + +**Entrypoint name:** ``memcached`` + +**Summary:** + +The memcached driver is a basic implementation and provides little +resiliency, though it's much simpler to setup. A lot of the features provided +in tooz are based on timeout (heartbeats, locks, etc) so are less resilient +than other backends. + +Considerations +~~~~~~~~~~~~~~ + +- Less resilient than other backends such as zookeeper and redis. +- Primitives are often based on TTL(s) that may expire before + being renewed. +- Lacks certain primitives (compare and delete) so certain functionality + is fragile and/or broken due to this. + +Redis +----- + +**Driver:** :py:class:`tooz.drivers.redis.RedisDriver` + +**Characteristics:** + +:py:attr:`tooz.drivers.redis.RedisDriver.CHARACTERISTICS` + +**Entrypoint name:** ``redis`` + +**Summary:** + +The redis driver is a basic implementation and provides reasonable resiliency +when used with `redis-sentinel`_. A lot of the features provided in tooz are +based on timeout (heartbeats, locks, etc) so are less resilient than other +backends. + +Considerations +~~~~~~~~~~~~~~ + +- Less resilient than other backends such as zookeeper. +- Primitives are often based on TTL(s) that may expire before + being renewed. + +IPC +--- + +**Driver:** :py:class:`tooz.drivers.ipc.IPCDriver` + +**Characteristics:** :py:attr:`tooz.drivers.ipc.IPCDriver.CHARACTERISTICS` + +**Entrypoint name:** ``ipc`` + +**Summary:** + +The IPC driver is based on Posix IPC API and implements a lock +mechanism and some basic group primitives (with **huge** +limitations). + +Considerations +~~~~~~~~~~~~~~ + +- The lock can **only** be distributed locally to a computer + processes. + +File +---- + +**Driver:** :py:class:`tooz.drivers.file.FileDriver` + +**Characteristics:** :py:attr:`tooz.drivers.file.FileDriver.CHARACTERISTICS` + +**Entrypoint name:** ``file`` + +**Summary:** + +The file driver is a **simple** driver based on files and directories. It +implements a lock based on POSIX or Window file level locking +mechanism and some basic group primitives (with **huge** +limitations). + +Considerations +~~~~~~~~~~~~~~ + +- The lock can **only** be distributed locally to a computer processes. +- Certain concepts provided by it are **not** crash tolerant. + +PostgreSQL +---------- + +**Driver:** :py:class:`tooz.drivers.pgsql.PostgresDriver` + +**Characteristics:** + +:py:attr:`tooz.drivers.pgsql.PostgresDriver.CHARACTERISTICS` + +**Entrypoint name:** ``postgresql`` + +**Summary:** + +The postgresql driver is a driver providing only a distributed lock (for now) +and is based on the `PostgreSQL database server`_ and its API(s) that provide +for `advisory locks`_ to be created and used by applications. When a lock is +acquired it will release either when explicitly released or automatically when +the database session ends (for example if the program using the lock crashes). + +Considerations +~~~~~~~~~~~~~~ + +- Lock that may be acquired restricted by + ``max_locks_per_transaction * (max_connections + max_prepared_transactions)`` + upper bound (PostgreSQL server configuration settings). + +MySQL +----- + +**Driver:** :py:class:`tooz.drivers.mysql.MySQLDriver` + +**Characteristics:** :py:attr:`tooz.drivers.mysql.MySQLDriver.CHARACTERISTICS` + +**Entrypoint name:** ``mysql`` + +**Summary:** + +The MySQL driver is a driver providing only distributed locks (for now) +and is based on the `MySQL database server`_ supported `get_lock`_ +primitives. When a lock is acquired it will release either when explicitly +released or automatically when the database session ends (for example if +the program using the lock crashes). + +Considerations +~~~~~~~~~~~~~~ + +- Does **not** work correctly on some MySQL versions. +- Does **not** work when MySQL replicates from one server to another (locks + are local to the server that they were created from). + +Etcd +---- + +**Driver:** :py:class:`tooz.drivers.etcd.EtcdDriver` + +**Characteristics:** :py:attr:`tooz.drivers.etcd.EtcdDriver.CHARACTERISTICS` + +**Entrypoint name:** ``etcd`` + +**Summary:** + +The etcd driver is a driver providing only distributed locks (for now) +and is based on the `etcd server`_ supported key/value storage and +associated primitives. + +Consul +------ + +**Driver:** :py:class:`tooz.drivers.consul.ConsulDriver` + +**Characteristics:** + +:py:attr:`tooz.drivers.consul.ConsulDriver.CHARACTERISTICS` + +**Entrypoint name:** ``consul`` + +**Summary:** + +The `consul`_ driver is a driver providing only distributed locks (for now) +and is based on the consul server key/value storage and/or +primitives. When a lock is acquired it will release either when explicitly +released or automatically when the consul session ends (for example if +the program using the lock crashes). + +Characteristics +--------------- + +.. autoclass:: tooz.coordination.Characteristics + +.. _etcd server: https://coreos.com/etcd/ +.. _consul: https://www.consul.io/ +.. _advisory locks: http://www.postgresql.org/docs/8.2/interactive/\ + explicit-locking.html#ADVISORY-LOCKS +.. _get_lock: http://dev.mysql.com/doc/refman/5.5/en/\ + miscellaneous-functions.html#function_get-lock +.. _PostgreSQL database server: http://postgresql.org +.. _MySQL database server: http://mysql.org +.. _redis-sentinel: http://redis.io/topics/sentinel diff --git a/doc/source/user/history.rst b/doc/source/user/history.rst new file mode 100644 index 0000000..f69be70 --- /dev/null +++ b/doc/source/user/history.rst @@ -0,0 +1 @@ +.. include:: ../../../ChangeLog diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst new file mode 100644 index 0000000..7ab2c1a --- /dev/null +++ b/doc/source/user/index.rst @@ -0,0 +1,15 @@ +================== +User Documentation +================== + +.. toctree:: + :maxdepth: 2 + + tutorial/index + drivers + compatibility + +.. toctree:: + :maxdepth: 1 + + history diff --git a/doc/source/user/tutorial/coordinator.rst b/doc/source/user/tutorial/coordinator.rst new file mode 100644 index 0000000..0131dee --- /dev/null +++ b/doc/source/user/tutorial/coordinator.rst @@ -0,0 +1,42 @@ +======================= + Creating A Coordinator +======================= + +The principal object provided by tooz is the *coordinator*. It allows you to +use various features, such as group membership, leader election or +distributed locking. + +The features provided by tooz coordinator are implemented using different +drivers. When creating a coordinator, you need to specify which back-end +driver you want it to use. Different drivers may provide different set of +capabilities. + +If a driver does not support a feature, it will raise a +:class:`~tooz.NotImplemented` exception. + +This example program loads a basic coordinator using the ZooKeeper based +driver. + +.. literalinclude:: ../../../../examples/coordinator.py + :language: python + +The second argument passed to the coordinator must be a unique identifier +identifying the running program. + +After the coordinator is created, it can be used to use the various features +provided. + +In order to keep the connection to the coordination server active, the method +:meth:`~tooz.coordination.CoordinationDriver.heartbeat` method must be called +regularly. This will ensure that the coordinator is not considered dead by +other program participating in the coordination. Unless you want to call it +manually, you can use tooz builtin heartbeat manager by passing the +`start_heart` argument. + +.. literalinclude:: ../../../../examples/coordinator_heartbeat.py + :language: python + +heartbeat at different moment or intervals. + +Note that certain drivers, such as `memcached` are heavily based on timeout, +so the interval used to run the heartbeat is important. diff --git a/doc/source/user/tutorial/group_membership.rst b/doc/source/user/tutorial/group_membership.rst new file mode 100644 index 0000000..502d211 --- /dev/null +++ b/doc/source/user/tutorial/group_membership.rst @@ -0,0 +1,41 @@ +===================== + Group Membership +===================== + +Basic operations +=================== + +One of the feature provided by the coordinator is the ability to handle +group membership. Once a group is created, any coordinator can join the +group and become a member of it. Any coordinator can be notified when a +members joins or leaves the group. + +.. literalinclude:: ../../../../examples/group_membership.py + :language: python + +Note that all the operation are asynchronous. That means you cannot be sure +that your group has been created or joined before you call the +:meth:`tooz.coordination.CoordAsyncResult.get` method. + +You can also leave a group using the +:meth:`tooz.coordination.CoordinationDriver.leave_group` method. The list of +all available groups is retrievable via the +:meth:`tooz.coordination.CoordinationDriver.get_groups` method. + +Watching Group Changes +====================== +It's possible to watch and get notified when the member list of a group +changes. That's useful to run callback functions whenever something happens +in that group. + + +.. literalinclude:: ../../../../examples/group_membership_watch.py + :language: python + +Using :meth:`tooz.coordination.CoordinationDriver.watch_join_group` and +:meth:`tooz.coordination.CoordinationDriver.watch_leave_group` your +application can be notified each time a member join or leave a group. To +stop watching an event, the two methods +:meth:`tooz.coordination.CoordinationDriver.unwatch_join_group` and +:meth:`tooz.coordination.CoordinationDriver.unwatch_leave_group` allow to +unregister a particular callback. diff --git a/doc/source/user/tutorial/hashring.rst b/doc/source/user/tutorial/hashring.rst new file mode 100644 index 0000000..1e00d1f --- /dev/null +++ b/doc/source/user/tutorial/hashring.rst @@ -0,0 +1,10 @@ +=========== + Hash ring +=========== + +Tooz provides a consistent hash ring implementation. It can be used to map +objects (represented via binary keys) to one or several nodes. When the node +list changes, the rebalancing of objects across the ring is kept minimal. + +.. literalinclude:: ../../../../examples/hashring.py + :language: python diff --git a/doc/source/user/tutorial/index.rst b/doc/source/user/tutorial/index.rst new file mode 100644 index 0000000..c2d398c --- /dev/null +++ b/doc/source/user/tutorial/index.rst @@ -0,0 +1,16 @@ +===================================== + Using Tooz in Your Application +===================================== + +This tutorial is a step-by-step walk-through demonstrating how to +use tooz in your application. + +.. toctree:: + :maxdepth: 2 + + coordinator + group_membership + leader_election + lock + hashring + partitioner diff --git a/doc/source/user/tutorial/leader_election.rst b/doc/source/user/tutorial/leader_election.rst new file mode 100644 index 0000000..df2a77b --- /dev/null +++ b/doc/source/user/tutorial/leader_election.rst @@ -0,0 +1,25 @@ +================= + Leader Election +================= + +Each group can elect its own leader. There can be only one leader at a time +in a group. Only members that are running for the election can be elected. +As soon as one of leader steps down or dies, a new member that was running +for the election will be elected. + +.. literalinclude:: ../../../../examples/leader_election.py + :language: python + +The method +:meth:`tooz.coordination.CoordinationDriver.watch_elected_as_leader` allows +to register for a function to be called back when the member is elected as a +leader. Using this function indicates that the run is therefore running for +the election. The member can stop running by unregistering all its callbacks +with :meth:`tooz.coordination.CoordinationDriver.unwatch_elected_as_leader`. +It can also temporarily try to step down as a leader with +:meth:`tooz.coordination.CoordinationDriver.stand_down_group_leader`. If +another member is in the run for election, it may be elected instead. + +To retrieve the leader of a group, even when not being part of the group, +the method :meth:`tooz.coordination.CoordinationDriver.get_leader()` can be +used. diff --git a/doc/source/user/tutorial/lock.rst b/doc/source/user/tutorial/lock.rst new file mode 100644 index 0000000..e3aaaae --- /dev/null +++ b/doc/source/user/tutorial/lock.rst @@ -0,0 +1,14 @@ +====== + Lock +====== + +Tooz provides distributed locks. A lock is identified by a name, and a lock can +only be acquired by one coordinator at a time. + +.. literalinclude:: ../../../../examples/lock.py + :language: python + +The method :meth:`tooz.coordination.CoordinationDriver.get_lock` allows +to create a lock identified by a name. Once you retrieve this lock, you can +use it as a context manager or use the :meth:`tooz.locking.Lock.acquire` and +:meth:`tooz.locking.Lock.release` methods to acquire and release the lock. diff --git a/doc/source/user/tutorial/partitioner.rst b/doc/source/user/tutorial/partitioner.rst new file mode 100644 index 0000000..24e12cf --- /dev/null +++ b/doc/source/user/tutorial/partitioner.rst @@ -0,0 +1,11 @@ +============= + Partitioner +============= + +Tooz provides a partitioner object based on its consistent hash ring +implementation. It can be used to map Python objects to one or several nodes. +The partitioner object automatically keeps track of nodes joining and leaving +the group, so the rebalancing is managed. + +.. literalinclude:: ../../../../examples/partitioner.py + :language: python diff --git a/tox.ini b/tox.ini index 2f5bcf2..49a0fb3 100644 --- a/tox.ini +++ b/tox.ini @@ -60,7 +60,7 @@ show-source = True [doc8] -ignore-path = doc/source/compatibility.rst,doc/source/history.rst +ignore-path = doc/source/user/compatibility.rst,doc/source/history.rst [testenv:releasenotes] -- cgit v1.2.1