summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* WIP: protocols/receivers:20200902-split-protos-8cWilly Tarreau2020-09-030-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's still not possible to eliminate the listener for UDP just because we don't store the I/O handler, the thread mask nor the context in the receiver. It doesn't make much sense to duplicate all these settings for each and every receiver for a given bind_conf. There seems to be something wrong with where the split is performed. Maybe we should first split the bind_conf to keep only receiver options. such as unix uid/gid/mode, interface, namespace, bind_thread etc. This would then become the real bind_conf, and the rest would be the listen_conf. The storage would look like this : proxy --+ | V listen_conf ---------> bind_conf ---> proto ||| ^ | VVV | V listeners ------------> receiver addr family Another option would be that the receivers are actually between the bind_conf and the listeners. We'd have this : proxy ---->>> bind_conf ---> proto ||| ||| ^ VVV VVV | listener <--> receivers The main issue is that we still need to support this syntax: bind ipv4@addr4:443,ipv6@addr6:443,quicv4@addr4:443,quicv6@addr6:443 ssl ... While it's acceptable to change the current behavior and have a single listener for all socket-based listeners (ipv4,ipv6,unix-stream etc), it's not when we switch to a new protocol (quic) as the setup will not be the same, and the receivers will be set up as UDP. So it *looks* like we need to keep multiple listeners per bind_conf. It looks like each listener will have a single receiver, even though this relation is not even mandatory for the long term. What matters instead is that the receiver knows how to find the listener and the settings that are common to the bind line regarding receivers: - interface - netns - unix perms - thread_mask proxy --+ | V bind_conf ------------> rx_settings ||| ^ VVV | listeners <------------ receiver ----> addr family | / +-----------> stream protocol -----'---> ctrl
* MEDIUM: protocol: do not call proto->bind() anymore from bind_listener()Willy Tarreau2020-09-024-39/+14
| | | | | | | All protocol's listeners now only take care of themselves and not of the receiver anymore since that's already being done in proto_bind_all(). Now it finally becomes obvious that UDP doesn't need a listener, as the only thing it does is to set the listener's state to LI_LISTEN!
* MEDIUM: protocol: explicitly start the receiver before the listenerWilly Tarreau2020-09-021-4/+41
| | | | | | | | | | Now protocol_bind_all() starts the receivers before their respective listeners so that ultimately we won't need the listeners for non- connected protocols. We still have to resort to an ugly trick to set the I/O handler in case of syslog over UDP because for now it's still not set in the receiver, so we hard-code it.
* MEDIUM: proto_sockpair: make use of sockpair_bind_receiver()Willy Tarreau2020-09-021-27/+9
| | | | | Now we rely on the address family's receiver instead of binding everything ourselves.
* MEDIUM: sockpair: implement sockpair_bind_receiver()Willy Tarreau2020-09-022-0/+61
| | | | | | | | | | | | | | Note that for now we don't have a sockpair.c file to host that unusual family, so the new function was placed directly into proto_sockpair.c. It's no big deal given that this family is currently not shared with multiple protocols. The function does almost nothing but setting up the receiver. This is normal as the socket the FDs are passed onto are supposed to have been already created somewhere else, and the only usable identifier for such a socket pair is the receiving FD itself. The function was assigned to sockpair's ->bind() and is not used yet.
* MEDIUM: uxst: make use of sock_unix_bind_receiver()Willy Tarreau2020-09-021-174/+16
| | | | | | | | | | | This removes all the AF_UNIX-specific code from uxst_bind_listener() and now simply relies on sock_unix_bind_listener() to do the same job. As mentionned in previous commit, the only difference is that now an unlikely failure on listen() will not result in a roll back of the temporary socket names since they will have been renamed during the bind() operation (as expected). But such failures do not correspond to any normal case and mostly denote operating system issues so there's no functionality loss here.
* MEDIUM: sock_unix: implement sock_unix_bind_receiver()Willy Tarreau2020-09-023-0/+211
| | | | | | | | | | | | | | This function performs all the bind-related stuff for UNIX sockets that was previously done in uxst_bind_listener(). There is a very tiny difference however, which is that previously, in the unlikely event where listen() would fail, it was still possible to roll back the binding and rename the backup to the original socket. Now we have to rename it before calling returning, hence it will be done before calling listen(). However, this doesn't cover any particular use case since listen() has no reason to fail there (and the rollback is not done for inherited sockets), that was just done that way as a generic error processing path. The code is not used yet and is referenced in the uxst proto's ->bind().
* MEDIUM: udp: make use of sock_inet_bind_receiver()Willy Tarreau2020-09-021-105/+19
| | | | | | | This removes all the AF_INET-specific code from udp_bind_listener() and now simply relies on sock_inet_bind_listener() to do the same job. The function is now basically just a wrapper around sock_inet_bind_receiver().
* MEDIUM: tcp: make use of sock_inet_bind_receiver()Willy Tarreau2020-09-021-121/+15
| | | | | | | This removes all the AF_INET-specific code from tcp_bind_listener() and now simply relies on sock_inet_bind_listener() to do the same job. The function was now roughly cut in half and its error path significantly simplified.
* MEDIUM: sock_inet: implement sock_inet_bind_receiver()Willy Tarreau2020-09-024-0/+150
| | | | | | | | | | | | | | | | | | | | | | | | This function collects all the receiver-specific code from both tcp_bind_listener() and udp_bind_listener() in order to provide a more generic AF_INET/AF_INET6 socket binding function. For now the API is not very elegant because some info are still missing from the receiver while there's no ideal place to fill them except when calling ->listen() at the protocol level. It looks like some polishing code is needed in check_config_validity() or somewhere around this in order to finalize the receivers' setup. The main issue is that listeners and receivers are created *before* bind_conf options are parsed and that there's no finishing step to resolve some of them. The function currently sets up a receiver and subscribes it to the poller. In an ideal world we wouldn't subscribe it but let the caller do it after having finished to configure the L4 stuff. The problem is that the caller would then need to perform an fd_insert() call and to possibly set the exported flag on the FD while it's not its job. Maybe an improvement could be to have a separate sock_start_receiver() call in sock.c. For now the function is not used but it will soon be. It's already referenced as tcp and udp's ->bind().
* MINOR: protocol: add a new ->bind() entry to bind the receiverWilly Tarreau2020-09-021-1/+2
| | | | | | This will be the function that must be used to bind the receiver. It solely depends on the address family but for now it's simpler to have it per protocol.
* MINOR: protocol: rename the ->bind field to ->listenWilly Tarreau2020-09-027-9/+9
| | | | | | The function currently is doing both the bind() and the listen(), so let's call it ->listen so that the bind() operation can move to another place.
* MINOR: sock: make sock_find_compatible_fd() only take a receiverWilly Tarreau2020-09-024-16/+16
| | | | | We don't need to have a listener anymore to find an fd, a receiver with its options properly set is enough now.
* MINOR: receiver: add the foreign and v6only flagsWilly Tarreau2020-09-023-10/+18
| | | | | | | | | The new RX_O_FOREIGN and RX_O_V6ONLY flags are now derived from the listener at the moment the ->bind() call is made. It's difficult to do it earlier because the keyword parsers are called one at a time without context so it's not possible to know whether v6only and v4v6 conflict for example, nor is it easy to know when all keywords were parsed. But fixing the receiver before starting it is fine.
* MINOR: listener: move the INHERITED flag down to the receiverWilly Tarreau2020-09-028-7/+8
| | | | | | It's the receiver's FD that's inherited from the parent process, not the listener's so the flag must move to the receiver so that appropriate actions can be taken.
* MINOR: receiver: add a receiver-specific flag to indicate the socket is boundWilly Tarreau2020-09-026-0/+24
| | | | | | | | In order to split the receiver from the listener, we'll need to know that a socket is already bound and ready to receive. We used to do that via tha LI_O_ASSIGNED state but that's not sufficient anymore since the receiver might not belong to a listener anymore. The new RX_O_BOUND flag is used for this
* MEDIUM: unix: move uid/gid/mode from bind_conf to receiverWilly Tarreau2020-09-027-21/+46
| | | | | | | | | | | | | | | Just like for the interface, these configuration elements are really receiver-specific and not just related to the bind conf, so they must move to the receiver. And just like for interfaces, this change required to iterate over all listeners when changing the parameter. The config supports default unix-bind settings, which used to be assigned to the bind_conf upon allocation. Now they're assigned to the receiver when a new listener is allocated. Side note: the bind_conf holds both bind parameters and listen parameters, in that some of these are properties of the listening socket and others of the newly instanciated sockets. Given that the vast majority is for the new sockets, it makes sense to move whatever remains to the receiver.
* REORG: listener: move the receiver part to a new fileWilly Tarreau2020-09-022-13/+53
| | | | | We'll soon add flags for the receivers, better add them to the final file, so it's time to move the definition to receiver-t.h.
* REORG: listeners: move the listener's proto to the receiverWilly Tarreau2020-09-0211-39/+39
| | | | | | | | The receiver is the one which depends on the protocol while the listener relies on the receiver. Let's move the protocol there. Since there's also a list element to get back to the listener from the proto list, this list element (proto_list) was moved as well. For now when scanning protos, we still see listeners which are linked by their rx.proto_list part.
* REORG: listeners: move the receiving FD to struct receiverWilly Tarreau2020-09-029-35/+33
| | | | | | | | | | | | The listening socket is represented by its file descriptor, which is generic to all receivers and not just listeners, so it must move to the rx struct. It's worth noting that in order to extend receivers and listeners to other protocols such as QUIC, we'll need other handles than file descriptors here, and that either a union or a cast to uintptr_t will have to be used. This was not done yet and the field was preserved under the name "fd" to avoid adding confusion.
* REORG: listeners: move the network interface to struct receiverWilly Tarreau2020-09-026-11/+11
| | | | | | | The interface is also an important discriminant for the address, so it belongs to the receiver. As a side note, the interface name is not freed upon exit.
* REORG: listeners: move the network namespace to struct receiverWilly Tarreau2020-09-028-21/+17
| | | | | The netns completes the address since the address' scope is valid within a namespace, so it must be stored with it.
* REORG: listeners: move the listening address to a struct receiverWilly Tarreau2020-09-0214-65/+65
| | | | The address will be specific to the receiver so let's move it there.
* REORG: listeners: create a new struct receiverWilly Tarreau2020-09-021-0/+6
| | | | | | | In order to start to split the listeners into the listener part and the event receiver part, we introduce a new field "rx" into struct listener that will eventually become a separate struct receiver. This patch only adds the struct with an options field that the receivers will need.
* MAJOR: init: start all listeners via protocols and not via proxies anymoreWilly Tarreau2020-09-025-75/+35
| | | | | | | | | | | | | | | | | | | | | | | | Ever since the protocols were added in 1.3.13, listeners used to be started twice: - once by start_proxies(), which iteratees over all proxies then all listeners ; - once by protocol_bind_all() which iterates over all protocols then all listeners ; It's a real mess because error reporting is not even consistent, and more importantly now that some protocols do not appear in regular proxies (peers, logs), there is no way to retry their binding should it fail on the last step. What this patch does is to make sure that listeners are exclusively started by protocols. The failure to start a listener now causes the emission of an error indicating the proxy's name (as it used to be the case per proxy), and retryable failures are silently ignored during all but last attempts. The start_proxies() function was kept solely for setting the proxy's state to READY and emitting the "Proxy started" message and log that some have likely got used to seeking in their logs.
* CLEANUP: protocol: remove all ->bind_all() and ->unbind_all() functionsWilly Tarreau2020-09-027-168/+2
| | | | | These ones were not used anymore since the two previous patches, let's drop them.
* MINOR: protocol: do not call proto->unbind_all() anymoreWilly Tarreau2020-09-021-3/+3
| | | | | | | | | | | | | | | Similarly to previous commit about ->bind_all(), we have the same construct for ->unbind_all() which ought not to be used either. Let's make protocol_unbind_all() iterate over all listeners and directly call unbind_listener() instead. It's worth noting that for uxst there was originally a specific ->unbind_all() function but the simplifications that came over the years have resulted in a locally reimplemented version of the same function: the test on (state > LI_ASSIGNED) there is equivalent to the one on (state >= LI_PAUSED) that is used in do_unbind_listener(), and it seems these have been equivalent since at least commit dabf2e264 ("[MAJOR] added a new state to listeners")) (1.3.14).
* MINOR: protocol: do not call proto->bind_all() anymoreWilly Tarreau2020-09-021-3/+5
| | | | | | | | | | All protocols only iterate over their own listeners list and start the listeners using a direct call to their ->bind() function. This code duplication doesn't make sense and prevents us from centralizing the startup error handling. Worse, it's not even symmetric because there's an unbind_all_listeners() function common to all protocols without any equivalent for binding. Let's start by directly calling each protocol's bind() function from protocol_bind_all().
* BUILD: thread: limit the libgcc_s workaround to glibc onlyWilly Tarreau2020-09-021-1/+1
| | | | | | | | | | Previous commit 77b98220e ("BUG/MINOR: threads: work around a libgcc_s issue with chrooting") broke the build on cygwin. I didn't even know we supported threads on cygwin. But the point is that it's actually the glibc-based libpthread which requires libgcc_s, so in absence of other reports we should not apply the workaround on other libraries. This should be backported along with the aforementioned patch.
* BUG/MINOR: threads: work around a libgcc_s issue with chrootingWilly Tarreau2020-09-021-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Sander Hoentjen reported another issue related to libgcc_s in issue #671. What happens is that when the old process quits, pthread_exit() calls something from libgcc_s.so after the process was chrooted, and this is the first call to that library, causing an attempt to load it. In a chroot, this fails, thus libthread aborts. The behavior widely differs between operating systems because some decided to use a static build for this library. In 2.2 this was resolved as a side effect of a workaround for the same issue with the backtrace() call, which is also in libgcc_s. This was in commit 0214b45 ("MINOR: debug: call backtrace() once upon startup"). But backtraces are not necessarily enabled, and we need something for older versions. By inspecting a significant number of ligcc_s on various gcc versions and platforms, it appears that a few functions have been present since gcc 3.0, one of which, _Unwind_Find_FDE() has no side effect (it only returns a pointer). What this patch does is that in the thread initialization code, if built with gcc >= 3.0, a call to this function is made in order to make sure that libgcc_s is loaded at start up time and that there will be no need to load it upon exit. An easy way to check which libs are loaded under Linux is : $ strace -e trace=openat ./haproxy -v With this patch applied, libgcc_s now appears during init. Sander confirmed that this patch was enough to put an end to the core dumps on exit in 2.0, so this patch should be backported there, and maybe as far as 1.8.
* REGTEST: increase some short timeouts to make tests more reliableWilly Tarreau2020-09-024-15/+15
| | | | | | | | | | | | | | | | | A few regtests continue to regularly fail in highly loaded VMs because they have very short timeouts. Actually the goal of running with short timeouts was to make sure we do not uselessly wait during tests designed to trigger them, but these timeouts here are never supposed to fire at all, so they don't need to be kept in the 15-20ms range. They do not pose any issue on any regular machine, but VMs are often suffering from huge time jumps and cannot always produce responses in that short of a time. Just like with commit ce6fc25b1 ("REGTEST: increase timeouts on the seamless-reload test"), let's raise these short timeouts to 1 second. A few other ones remain set to 150-200ms and do not seem to cause any issue. Some are actually expected to trigger so let's not touch them for now.
* CLEANUP: http: silence a cppcheck warning in get_http_auth()Willy Tarreau2020-09-021-3/+3
| | | | | | | | | | | | | | | | | | | In issue #777, cppcheck wrongly assumes a useless null pointer check in the expression below while it's obvious that in a 3G/1G split on 32-bit, len can become positive if p is NULL: p = memchr(ctx.value.ptr, ' ', ctx.value.len); len = p - ctx.value.ptr; if (!p || len <= 0) return 0; In addition, on 64 bits you never know given that len is a 32-bit signed int thus the sign of the result in case of a null p will always be the opposite of the 32th bit of ctx.value.ptr. Admittedly the test is ugly. Tim proposed this fix consisting in checking for p == ctx.value.ptr instead when checking for first character only, which Ilya confirmed is enough to shut cppcheck up. No backport is needed.
* BUG/MEDIUM: contrib/spoa-server: Fix ipv4_address used instead of ipv6_addressGilchrist Dadaglo2020-09-011-1/+1
| | | | | | Typo leads to checking the wrong object for memory issues This patch must be backported as far as 2.0.
* BUG/MINOR: contrib/spoa-server: Updating references to free in case of failureGilchrist Dadaglo2020-09-011-0/+9
| | | | | | | When we encounter a failure, all previously borrowed references should be freed. Especially if the program is not failing immediately This patch must be backported as far as 2.0.
* BUG/MINOR: contrib/spoa-server: Do not free reference to NULLGilchrist Dadaglo2020-09-011-1/+0
| | | | | | | As per https://docs.python.org/3/c-api/refcounting.html, Py_DECREF should not be called on NULL objects. This patch must be backported as far as 2.0.
* BUG/MINOR: contrib/spoa-server: Ensure ip address references are freedGilchrist Dadaglo2020-09-011-0/+4
| | | | | | | | IP addresses references passed in argument for ps_python are not freed after they have been used. Leading to a small chance of mem leak if a lot of ip addresses are passed around This patch must be backported as far as 2.0.
* BUG/MAJOR: contrib/spoa-server: Fix unhandled python call leading to memory leakGilchrist Dadaglo2020-09-011-0/+4
| | | | | | | | | | | The result from spoa evaluation of the user provided python code is never passed back to the main spoa process nor freed. Same for the keyword list passed. This results into the elements never freed by Python as reference count never goes down. https://docs.python.org/3/extending/extending.html#reference-counting-in-python This patch must be backported as far as 2.0.
* MINOR: contrib/spoa-server: allow MAX_FRAME_SIZE overrideBertrand Jacquin2020-09-011-0/+3
| | | | | MAX_FRAME_SIZE is forced to the default value of tune.bufsize, however they don't necessarily have to be tight together.
* MINOR: http-htx: Handle an optional reason when replacing the response statusChristopher Faulet2020-09-013-10/+10
| | | | | | | | | | When calling the http_replace_res_status() function, an optional reason may now be set. It is ignored if it points to NULL and the original reason is preserved. Only the response status is replaced. Otherwise both the status and the reason are replaced. It simplifies the API and most of time, avoids an extra call to http_replace_res_reason().
* BUG/MINOR: http-rules: Replace path and query-string in "replace-path" actionChristopher Faulet2020-09-012-3/+8
| | | | | | | | | | | | | | | The documentation stated the "replace-path" action replaces the path, including the query-string if any is present. But in the code, only the path is replaced. The query-string is preserved. So, now, instead of relying on the same action code than "set-uri" action (1), a new action code (4) is used for "replace-path" action. In http_req_replace_stline() function, when the action code is 4, we call http_replace_req_path() setting the last argument (with_qs) to 1. This way, the query-string is not skipped but included to the path to be replaced. This patch relies on the commit b8ce505c6 ("MINOR: http-htx: Add an option to eval query-string when the path is replaced"). Both must be backported as far as 2.0. It should fix the issue #829.
* MINOR: http-htx: Add an option to eval query-string when the path is replacedChristopher Faulet2020-09-013-7/+13
| | | | | | | | | The http_replace_req_path() function now takes a third argument to evaluate the query-string as part of the path or to preserve it. If <with_qs> is set, the query-string is replaced with the path. Otherwise, only the path is replaced. This patch is mandatory to fix issue #829. The next commit depends on it. So be carefull during backports.
* BUG/MEDIUM: http-ana: Don't wait to send 1xx responses received from serversChristopher Faulet2020-08-311-0/+1
| | | | | | | | | | | | | | | | | | When an informational response (1xx) is received, we must be sure to send it ASAP. To do so, CF_SEND_DONTWAIT flag must be set on the response channel to instruct the stream-interface to not set the CO_SFL_MSG_MORE flag on the transport layer. Otherwise the response delivery may be delayed, because of the commit 8945bb6c0 ("BUG/MEDIUM: stream-int: fix loss of CO_SFL_MSG_MORE flag in forwarding"). Note that a previous patch (cf6898cd ["BUG/MINOR: http-ana: Don't wait to send 1xx responses generated by HAProxy"]) add this flag on 1xx responses generated by HAProxy but not on responses coming from servers. This patch must be backported to 2.2 and may be backported as far as 1.9, for HTX part only. But this part has changed in the 2.2, so it may be a bit tricky. Note it does not fix any known bug on 2.1 and below because the CO_SFL_MSG_MORE flag is ignored by the h1 mux.
* BUILD: sock_unix: fix build issue with isdigit()Willy Tarreau2020-08-291-2/+2
| | | | | | | Commit 0d06df6 ("MINOR: sock: introduce sock_inet and sock_unix") made use of isdigit() on the UNIX socket path without casting the value to unsigned char, breaking the build on cygwin and possibly other platforms. No backport is needed.
* MINOR: sock: distinguish dgram from stream types when retrieving old socketsWilly Tarreau2020-08-282-0/+11
| | | | | | | | | For now we still don't retrieve dgram sockets, but the code must be able to distinguish them before we switch to receivers. This adds a new flag to the xfer_sock_list indicating that a socket is of type SOCK_DGRAM. The way to set the flag for now is by looking at the dummy address family which equals AF_CUST_UDP{4,6} in this case (given that other dgram sockets are not yet supported).
* MINOR: sock: do not use LI_O_* in xfer_sock_list anymoreWilly Tarreau2020-08-282-10/+14
| | | | | | We'll want to store more info there and some info that are not represented in listener options at the moment (such as dgram vs stream) so let's get rid of these and instead use a new set of options (SOCK_XFER_OPT_*).
* REORG: sock: move get_old_sockets() from haproxy.cWilly Tarreau2020-08-283-240/+264
| | | | | | | The new function was called sock_get_old_sockets() and was left as-is except a minimum amount of style lifting to make it more readable. It will never be awesome anyway since it's used very early in the boot sequence and needs to perform socket I/O without any external help.
* MINOR: sock_inet: move the IPv4/v6 transparent mode code to sock_inetWilly Tarreau2020-08-284-105/+77
| | | | | | This code was highly redundant, existing for TCP clients, TCP servers and UDP servers. Let's move it to sock_inet where it belongs. The new functions are sock_inet4_make_foreign() and sock_inet6_make_foreign().
* MINOR: sock: implement sock_find_compatible_fd()Willy Tarreau2020-08-284-91/+72
| | | | | | | | | | | | | | | | | This is essentially a merge from tcp_find_compatible_fd() and uxst_find_compatible_fd() that relies on a listener's address and compare function and still checks for other variations. For AF_INET6 it compares a few of the listener's bind options. A minor change for UNIX sockets is that transparent mode, interface and namespace used to be ignored when trying to pick a previous socket while now if they are changed, the socket will not be reused. This could be refined but it's still better this way as there is no more risk of using a differently bound socket by accident. Eventually we should not pass a listener there but a set of binding parameters (address, interface, namespace etc...) which ultimately will be grouped into a receiver. For now this still doesn't exist so let's stick to the listener to break dependencies in the rest of the code.
* MINOR: sock: add interface and namespace length to xfer_sock_listWilly Tarreau2020-08-282-0/+4
| | | | This will ease and speed up comparisons in FD lookups.
* REORG: listener: move xfer_sock_list to sock.{c,h}.Willy Tarreau2020-08-287-15/+55
| | | | | This will be used for receivers as well thus it is not specific to listeners but to sockets.
View Lorry Controller Status