diff options
Diffstat (limited to 'doc/source/admin/zmq_driver.rst')
-rw-r--r-- | doc/source/admin/zmq_driver.rst | 602 |
1 files changed, 602 insertions, 0 deletions
diff --git a/doc/source/admin/zmq_driver.rst b/doc/source/admin/zmq_driver.rst new file mode 100644 index 0000000..88ccf8a --- /dev/null +++ b/doc/source/admin/zmq_driver.rst @@ -0,0 +1,602 @@ +------------------------------ +ZeroMQ Driver Deployment Guide +------------------------------ + +.. currentmodule:: oslo_messaging + +============ +Introduction +============ + +0MQ (also known as ZeroMQ or zmq) is embeddable networking library +but acts like a concurrency framework. It gives you sockets +that carry atomic messages across various transports +like in-process, inter-process, TCP, and multicast. You can connect +sockets N-to-N with patterns like fan-out, pub-sub, task distribution, +and request-reply. It's fast enough to be the fabric for clustered +products. Its asynchronous I/O model gives you scalable multi-core +applications, built as asynchronous message-processing tasks. It has +a score of language APIs and runs on most operating systems. + +Originally the zero in 0MQ was meant as "zero broker" and (as close to) +"zero latency" (as possible). Since then, it has come to encompass +different goals: zero administration, zero cost, and zero waste. +More generally, "zero" refers to the culture of minimalism that permeates +the project. + +More detail regarding ZeroMQ library is available from the `specification`_. + +.. _specification: http://zguide.zeromq.org/page:all + +======== +Abstract +======== + +Currently, ZeroMQ is one of the RPC backend drivers in oslo.messaging. ZeroMQ +can be the only RPC driver across the OpenStack cluster. +This document provides deployment information for this driver in oslo_messaging. + +Other than AMQP-based drivers, like RabbitMQ, default ZeroMQ doesn't have +any central brokers in oslo.messaging, instead, each host (running OpenStack +services) is both ZeroMQ client and server. As a result, each host needs to +listen to a certain TCP port for incoming connections and directly connect +to other hosts simultaneously. + +Another option is to use a router proxy. It is not a broker because it +doesn't assume any message ownership or persistence or replication etc. It +performs only a redirection of messages to endpoints taking routing info from +message envelope. + +Topics are used to identify the destination for a ZeroMQ RPC call. There are +two types of topics, bare topics and directed topics. Bare topics look like +'compute', while directed topics look like 'compute.machine1'. + +======== +Scenario +======== + +Assuming the following systems as a goal. + +:: + + +--------+ + | Client | + +----+---+ + | + -----+---------+-----------------------+--------------------- + | | + +--------+------------+ +-------+----------------+ + | Controller Node | | Compute Node | + | Nova | | Neutron | + | Keystone | | Nova | + | Glance | | nova-compute | + | Neutron | | Ceilometer | + | Cinder | | | + | Ceilometer | +------------------------+ + | zmq-proxy | + | Redis | + | Horizon | + +---------------------+ + + +=================== +Basic Configuration +=================== + +Enabling (mandatory) +-------------------- + +To enable the driver the 'transport_url' option must be set to 'zmq://' +in the section [DEFAULT] of the conf file, the 'rpc_zmq_host' option +must be set to the hostname of the current node. :: + + [DEFAULT] + transport_url = "zmq://" + + [oslo_messaging_zmq] + rpc_zmq_host = {hostname} + +Default configuration of zmq driver is called 'Static Direct Connections' (To +learn more about zmq driver configurations please proceed to the corresponding +section 'Existing Configurations'). That means that all services connect +directly to each other and all connections are static so we open them at the +beginning of service's lifecycle and close them only when service quits. This +configuration is the simplest one since it doesn't require any helper services +(proxies) other than matchmaker to be running. + + +Matchmaking (mandatory) +----------------------- + +The ZeroMQ driver implements a matching capability to discover hosts available +for communication when sending to a bare topic. This allows broker-less +communications. + +The Matchmaker is pluggable and it provides two different Matchmaker classes. + +MatchmakerDummy: default matchmaker driver for all-in-one scenario (messages +are sent to itself; used mainly for testing). + +MatchmakerRedis: loads the hash table from a remote Redis server, supports +dynamic host/topic registrations, host expiration, and hooks for consuming +applications to acknowledge or neg-acknowledge topic.host service availability. + +For ZeroMQ driver Redis is configured in transport_url also. For using Redis +specify the URL as follows:: + + [DEFAULT] + transport_url = "zmq+redis://127.0.0.1:6379" + +In order to cleanup redis storage from expired records (e.g. target listener +goes down) TTL may be applied for keys. Configure 'zmq_target_expire' option +which is 300 (seconds) by default. The option is related not specifically to +redis so it is also defined in [oslo_messaging_zmq] section. If option value +is <= 0 then keys don't expire and live forever in the storage. + +The other option is 'zmq_target_update' (180 seconds by default) which +specifies how often each RPC-Server should update the matchmaker. This option's +optimal value generally is zmq_target_expire / 2 (or 1.5). It is recommended to +calculate it based on 'zmq_target_expire' so services records wouldn't expire +earlier than being updated from alive services. + +Generally matchmaker can be considered as an alternate approach to services +heartbeating. + + +Matchmaker Data Source (mandatory) +---------------------------------- + +Matchmaker data source is stored in files or Redis server discussed in the +previous section. How to make up the database is the key issue for making ZeroMQ +driver work. + +If deploying the MatchmakerRedis, a Redis server is required. Each (K, V) pair +stored in Redis is that the key is a base topic and the corresponding values are +hostname arrays to be sent to. + + +HA for Redis database +--------------------- + +Single node Redis works fine for testing, but for production there is some +availability guarantees wanted. Without Redis database zmq deployment should +continue working anyway, because there is no need in Redis for services when +connections established already. But if you would like to restart some services +or run more workers or add more hardware nodes to the deployment you will need +nodes discovery mechanism to work and it requires Redis. + +To provide database recovery in situations when redis node goes down for example, +we use Sentinel solution and redis master-slave-slave configuration (if we have +3 controllers and run Redis on each of them). + +To deploy redis with HA follow the `sentinel-install`_ instructions. From the +messaging driver's side you will need to setup following configuration :: + + [DEFAULT] + transport_url = "zmq+sentinel://host1:26379,host2:26379,host3:26379" + + +Listening Address (optional) +---------------------------- + +All services bind to an IP address or Ethernet adapter. By default, all services +bind to '*', effectively binding to 0.0.0.0. This may be changed with the option +'rpc_zmq_bind_address' which accepts a wildcard, IP address, or Ethernet adapter. + +This configuration can be set in [oslo_messaging_zmq] section. + +For example:: + + rpc_zmq_bind_address = * + +Currently zmq driver uses dynamic port binding mechanism, which means that +each listener will allocate port of a random number (static, i.e. fixed, ports +may only be used for sockets inside proxies now). Ports range is controlled +by two options 'rpc_zmq_min_port' and 'rpc_zmq_max_port'. Change them to +restrict current service's port binding range. 'rpc_zmq_bind_port_retries' +controls number of retries before 'ports range exceeded' failure. + +For example:: + + rpc_zmq_min_port = 49153 + rpc_zmq_max_port = 65536 + rpc_zmq_bind_port_retries = 100 + + +======================= +Existing Configurations +======================= + + +Static Direct Connections +------------------------- + +The example of service config file:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = false + use_router_proxy = false + use_dynamic_connections = false + zmq_target_expire = 60 + zmq_target_update = 30 + rpc_zmq_min_port = 49153 + rpc_zmq_max_port = 65536 + +In both static and dynamic direct connections configuration it is necessary to +configure firewall to open binding port range on each node:: + + iptables -A INPUT -p tcp --match multiport --dports 49152:65535 -j ACCEPT + + +The sequrity recommendation here (it is general for any RPC backend) is to +setup private network for message bus and another open network for public APIs. +ZeroMQ driver doesn't support authentication and encryption on its level. + +As stated above this configuration is the simplest one since it requires only a +Matchmaker service to be running. That is why driver's options configured by +default in a way to use this type of topology. + +The biggest advantage of static direct connections (other than simplicity) is +it's huge performance. On small deployments (20 - 50 nodes) it can outperform +brokered solutions (or solutions with proxies) 3x - 5x times. It becomes possible +because this configuration doesn't have a central node bottleneck so it's +throughput is limited by only a TCP and network bandwidth. + +Unfortunately this approach can not be applied as is on a big scale (over 500 nodes). +The main problem is the number of connections between services and particularly +the number of connections on each controller node grows (in a worst case) as +a square function of number of the whole running services. That's not +appropriate. + +However this approach can be successfully used and is recommended to be used +when services on controllers doesn't talk to agent services on resource nodes +using oslo.messaging RPC, but RPC is used only to communicate controller +services between each other. + +Examples here may be Cinder+Ceph backend and Ironic how it utilises +oslo.messaging. + +For all the other cases like Nova and Neutron on a big scale using proxy-based +configurations or dynamic connections configuration is more appropriate. + +The exception here may be the case when using OpenStack services inside Docker +containers with Kubernetes. Since Kubernetes already solves similar problems by +using KubeProxy and virtual IP addresses for each container. So it manages all +the traffic using iptables which is more than appropriate to solve the problem +described above. + +Summing up it is recommended to use this type of zmq configuration for + +1. Small clouds (up to 100 nodes) +2. Cinder+Ceph deployment +3. Ironic deployment +4. OpenStack + Kubernetes (OpenStack in containers) deployment + + +Dynamic Direct Connections +-------------------------- +The example of service config file:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = false + use_router_proxy = false + + use_dynamic_connections = true + zmq_failover_connections = 2 + zmq_linger = 60 + + zmq_target_expire = 60 + zmq_target_update = 30 + rpc_zmq_min_port = 49153 + rpc_zmq_max_port = 65536 + +The 'use_dynamic_connections = true' obviously states that connections are dynamic. +'zmq_linger' become crucial with dynamic connections in order to avoid socket +leaks. If socket being connected to a wrong (dead) host which somehow still +present in the Matchmaker and message was sent, then the socket can not be closed +until message stays in the queue (the default linger is infinite waiting). So +need to specify linger explicitly. + +Services often run more than one worker on the same topic. Workers are equal, so +any can handle the message. In order to connect to more than one available worker +need to setup 'zmq_failover_connections' option to some value (2 by default which +means 2 additional connections). Take care because it may also result in slow-down. + +All recommendations regarding port ranges described in previous section are also +valid here. + +Most things are similar to what we had with static connections the only +difference is that each message causes connection setup and disconnect afterwards +immediately after message was sent. + +The advantage of this deployment is that average number of connections on +controller node at any moment is not high even for quite large deployments. + +The disadvantage is overhead caused by need to connect/disconnect per message. +So this configuration can with no doubt be considered as the slowest one. The +good news is the RPC of OpenStack doesn't require "thousands message per second" +bandwidth per each particular service (do not confuse with central broker/proxy +bandwidth which is needed as high as possible for a big scale and can be a +serious bottleneck). + +One more bad thing about this particular configuration is fanout. Here it is +completely linear complexity operation and it suffers the most from +connect/disconnect overhead per message. So for fanout it is fair to say that +services can have significant slow-down with dynamic connections. + +The recommended way to solve this problem is to use combined solution with +proxied PUB/SUB infrastructure for fanout and dynamic direct connections for +direct message types (plain CAST and CALL messages). This combined approach +will be described later in the text. + + +Router Proxy +------------ + +The example of service config file:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = false + use_router_proxy = true + use_dynamic_connections = false + +The example of proxy config file:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = false + + [zmq_proxy_opt] + host = host-1 + +RPC may consume too many TCP sockets on controller node in directly connected +configuration. To solve the issue ROUTER proxy may be used. + +In order to configure driver to use ROUTER proxy set up the 'use_router_proxy' +option to true in [oslo_messaging_zmq] section (false is set by default). + +Pay attention to 'use_pub_sub = false' line, which has to match for all +services and proxies configs, so it wouldn't work if proxy uses PUB/SUB and +services don't. + +Not less than 3 proxies should be running on controllers or on stand alone +nodes. The parameters for the script oslo-messaging-zmq-proxy should be:: + + oslo-messaging-zmq-proxy + --config-file /etc/oslo/zeromq.conf + --log-file /var/log/oslo/zeromq-router-proxy.log + --host node-123 + --frontend-port 50001 + --backend-port 50002 + --debug + +Config file for proxy consists of default section, 'oslo_messaging_zmq' section +and additional 'zmq_proxy_opts' section. + +Command line arguments like host, frontend_port, backend_port and publisher_port +respectively can also be set in 'zmq_proxy_opts' section of a configuration +file (i.e., /etc/oslo/zeromq.conf). All arguments are optional. + +Port value of 0 means random port (see the next section for more details). + +Take into account that --debug flag makes proxy to make a log record per every +dispatched message which influences proxy performance significantly. So it is +not recommended flag to use in production. Without --debug there will be only +Matchmaker updates or critical errors in proxy logs. + +In this configuration we use proxy as a very simple dispatcher (so it has the +best performance with minimal overhead). The only thing proxy does is getting +binary routing-key frame from the message and dispatch message on this key. + +In this kind of deployment client is in charge of doing fanout. Before sending +fanout message client takes a list of available hosts for the topic and sends +as many messages as the number of hosts it got. + +This configuration just uses DEALER/ROUTER pattern of ZeroMQ and doesn't use +PUB/SUB as it was stated above. + +Disadvantage of this approach is again slower client fanout. But it is much +better than with dynamic direct connections because we don't need to connect +and disconnect per each message. + + +ZeroMQ PUB/SUB Infrastructure +----------------------------- + +The example of service config file:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = true + use_router_proxy = true + use_dynamic_connections = false + +The example of proxy config file:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = true + + [zmq_proxy_opt] + host = host-1 + +It seems obvious that fanout pattern of oslo.messaging maps on ZeroMQ PUB/SUB +pattern, but it is only at first glance. It does really, but lets look a bit +closer. + +First caveat is that in oslo.messaging it is a client who makes fanout (and +generally initiates conversation), server is passive. While in ZeroMQ publisher +is a server and subscribers are clients. And here is the problem: RPC-servers +are subscribers in terms of ZeroMQ PUB/SUB, they hold the SUB socket and wait +for messages. And they don't know anything about RPC-clients, and clients +generally come later than servers. So servers don't have a PUB to subscribe +on start, so we need to introduce something in the middle, and here the proxy +plays the role. + +Publisher proxy has ROUTER socket on the front-end and PUB socket on the back-end. +So client connects to ROUTER and sends a single message to a publisher proxy. +Proxy redirects this message to PUB socket which performs actual publishing. + +Command to run central publisher proxy:: + + oslo-messaging-zmq-proxy + --config-file /etc/oslo/zeromq.conf + --log-file /var/log/oslo/zeromq-router-proxy.log + --host node-123 + --frontend-port 50001 + --publisher-port 50003 + --debug + +When we run a publisher proxy we need to specify a --publisher-port option. +Random port will be picked up otherwise and clients will get it from the +Matchmaker. + +The advantage of this approach is really fast fanout, while it takes time on +proxy to publish, but ZeroMQ PUB/SUB is one of the fastest fanout pattern +implementations. It also makes clients faster, because they need to send only a +single message to a proxy. + +In order to balance load and HA it is recommended to have at least 3 proxies basically, +but the number of running proxies is not limited. They also don't form a cluster, +so there are no limitations on number caused by consistency algorithm requirements. + +The disadvantage is that number of connections on proxy increased twice compared +to previous deployment, because we still need to use router for direct messages. + +The documented limitation of ZeroMQ PUB/SUB is 10k subscribers. + +In order to limit the number of subscribers and connections the local proxies +may be used. In order to run local publisher the following command may be used:: + + + oslo-messaging-zmq-proxy + --local-publisher + --config-file /etc/oslo/zeromq.conf + --log-file /var/log/oslo/zeromq-router-proxy.log + --host localhost + --publisher-port 60001 + --debug + +Pay attention to --local-publisher flag which specifies the type of a proxy. +Local publishers may be running on every single node of a deployment. To make +services use of local publishers the 'subscribe_on' option has to be specified +in service's config file:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = true + use_router_proxy = true + use_dynamic_connections = false + subscribe_on = localhost:60001 + +If we forgot to specify the 'subscribe_on' services will take info from Matchmaker +and still connect to a central proxy, so the trick wouldn't work. Local proxy +gets all the needed info from the matchmaker in order to find central proxies +and subscribes on them. Frankly speaking you can pub a central proxy in the +'subscribe_on' value, even a list of hosts may be passed the same way as we do +for the transport_url:: + + subscribe_on = host-1:50003,host-2:50003,host-3:50003 + +This is completely valid, just not necessary because we have information about +central proxies in Matchmaker. One more thing to highlight about 'subscribe_on' +is that it has higher priority than Matchmaker if being explicitly mentioned. + +Concluding all the above, fanout over PUB/SUB proxies is the best choice +because of static connections infrastructure, fail over when one or some publishers +die, and ZeroMQ PUB/SUB high performance. + + +What If Mix Different Configurations? +------------------------------------- + +Three boolean variables 'use_pub_sub', 'use_router_proxy' and 'use_dynamic_connections' +give us exactly 8 possible combinations. But from practical perspective not all +of them are usable. So lets discuss only those which make sense. + +The main recommended combination is Dynamic Direct Connections plus PUB/SUB +infrastructure. So we deploy PUB/SUB proxies as described in corresponding +paragraph (either with local+central proxies or with only a central proxies). +And the services configuration file will look like the following:: + + [DEFAULT] + transport_url = "zmq+redis://host-1:6379" + + [oslo_messaging_zmq] + use_pub_sub = true + use_router_proxy = false + use_dynamic_connections = true + +So we just tell the driver not to pass direct messages CALL and CAST over router, +but send them directly to RPC servers. All the details of configuring services +and port ranges has to be taken from 'Dynamic Direct Connections' paragraph. +So it's combined configuration. Currently it is the best choice from number of +connections perspective. + +Frankly speaking, deployment from the 'ZeroMQ PUB/SUB Infrastructure' section is +also a combination of 'Router Proxy' with PUB/SUB, we've just used the same +proxies for both. + +Here we've discussed combination inside the same service. But configurations can +also be combined on a higher level, a level of services. So you could have for +example a deployment where Cinder uses static direct connections and Nova/Neutron +use combined PUB/SUB + dynamic direct connections. But such approach needs additional +caution and may be confusing for cloud operators. Still it provides maximum +optimization of performance and number of connections on proxies and controller +nodes. + + +================ +DevStack Support +================ + +ZeroMQ driver can be tested on a single node deployment with DevStack. Take +into account that on a single node it is not that obvious any performance +increase compared to other backends. To see significant speed up you need at least +20 nodes. + +In local.conf [localrc] section need to enable zmq plugin which lives in +`devstack-plugin-zmq`_ repository. + +For example:: + + enable_plugin zmq https://github.com/openstack/devstack-plugin-zmq.git + + +Example of local.conf:: + + [[local|localrc]] + DATABASE_PASSWORD=password + ADMIN_PASSWORD=password + SERVICE_PASSWORD=password + SERVICE_TOKEN=password + + enable_plugin zmq https://github.com/openstack/devstack-plugin-zmq.git + + OSLOMSG_REPO=https://review.openstack.org/openstack/oslo.messaging + OSLOMSG_BRANCH=master + + ZEROMQ_MATCHMAKER=redis + LIBS_FROM_GIT=oslo.messaging + ENABLE_DEBUG_LOG_LEVEL=True + + +.. _devstack-plugin-zmq: https://github.com/openstack/devstack-plugin-zmq.git +.. _sentinel-install: http://redis.io/topics/sentinel |