curl_multi_poll: a sister to curl_multi_wait() that waits more

Repeatedly we see problems where using curl_multi_wait() is difficult or
just awkward because if it has no file descriptor to wait for
internally, it returns immediately and leaves it to the caller to wait
for a small amount of time in order to avoid occasional busy-looping.

This is often missed or misunderstood, leading to underperforming
applications.

This change introduces curl_multi_poll() as a replacement drop-in
function that accepts the exact same set of arguments. This function
works identically to curl_multi_wait() - EXCEPT - for the case when
there's nothing to wait for internally, as then this function will by
itself wait for a "suitable" short time before it returns. This
effectiely avoids all risks of busy-looping and should also make it less
likely that apps "over-wait".

This also changes the curl tool to use this funtion internally when
doing parallel transfers and changes curl_easy_perform() to use it
internally.

Closes #4163

2 files changed, 111 insertions, 0 deletions

diff --git a/docs/libcurl/Makefile.inc b/docs/libcurl/Makefile.inc
index b4ff45dde..bd88c9c38 100644
--- a/docs/libcurl/Makefile.inc
+++ b/docs/libcurl/Makefile.inc
@@ -46,6 +46,7 @@ man_MANS = \
  curl_multi_info_read.3 \
  curl_multi_init.3 \
  curl_multi_perform.3 \
+ curl_multi_poll.3 \
  curl_multi_remove_handle.3 \
  curl_multi_setopt.3 \
  curl_multi_socket.3 \
diff --git a/docs/libcurl/curl_multi_poll.3 b/docs/libcurl/curl_multi_poll.3
new file mode 100644
index 000000000..9fc72c55d
--- /dev/null
+++ b/docs/libcurl/curl_multi_poll.3
@@ -0,0 +1,110 @@
+.\" **************************************************************************
+.\" *                                  _   _ ____  _
+.\" *  Project                     ___| | | |  _ \| |
+.\" *                             / __| | | | |_) | |
+.\" *                            | (__| |_| |  _ <| |___
+.\" *                             \___|\___/|_| \_\_____|
+.\" *
+.\" * Copyright (C) 1998 - 2019, Daniel Stenberg, <daniel@haxx.se>, et al.
+.\" *
+.\" * This software is licensed as described in the file COPYING, which
+.\" * you should have received as part of this distribution. The terms
+.\" * are also available at https://curl.haxx.se/docs/copyright.html.
+.\" *
+.\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell
+.\" * copies of the Software, and permit persons to whom the Software is
+.\" * furnished to do so, under the terms of the COPYING file.
+.\" *
+.\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
+.\" * KIND, either express or implied.
+.\" *
+.\" **************************************************************************
+.TH curl_multi_poll 3 "29 Jul 2019" "libcurl 7.66.0" "libcurl Manual"
+.SH NAME
+curl_multi_poll - polls on all easy handles in a multi handle
+.SH SYNOPSIS
+.nf
+#include <curl/curl.h>
+
+CURLMcode curl_multi_poll(CURLM *multi_handle,
+                          struct curl_waitfd extra_fds[],
+                          unsigned int extra_nfds,
+                          int timeout_ms,
+                          int *numfds);
+.ad
+.SH DESCRIPTION
+\fIcurl_multi_poll(3)\fP polls all file descriptors used by the curl easy
+handles contained in the given multi handle set.  It will block until activity
+is detected on at least one of the handles or \fItimeout_ms\fP has passed.
+Alternatively, if the multi handle has a pending internal timeout that has a
+shorter expiry time than \fItimeout_ms\fP, that shorter time will be used
+instead to make sure timeout accuracy is reasonably kept.
+
+The calling application may pass additional curl_waitfd structures which are
+similar to \fIpoll(2)\fP's pollfd structure to be waited on in the same call.
+
+On completion, if \fInumfds\fP is non-NULL, it will be populated with the
+total number of file descriptors on which interesting events occurred. This
+number can include both libcurl internal descriptors as well as descriptors
+provided in \fIextra_fds\fP.
+
+If no extra file descriptors are provided and libcurl has no file descriptor
+to offer to wait for, this function will instead wait during \fItimeout_ms\fP
+milliseconds (or shorter if an internal timer indicates so). This is the
+detail that makes this function different than \fIcurl_multi_wait(3)\fP.
+
+This function is encouraged to be used instead of select(3) when using the
+multi interface to allow applications to easier circumvent the common problem
+with 1024 maximum file descriptors.
+.SH curl_waitfd
+.nf
+struct curl_waitfd {
+  curl_socket_t fd;
+  short events;
+  short revents;
+};
+.fi
+.IP CURL_WAIT_POLLIN
+Bit flag to curl_waitfd.events indicating the socket should poll on read
+events such as new data received.
+.IP CURL_WAIT_POLLPRI
+Bit flag to curl_waitfd.events indicating the socket should poll on high
+priority read events such as out of band data.
+.IP CURL_WAIT_POLLOUT
+Bit flag to curl_waitfd.events indicating the socket should poll on write
+events such as the socket being clear to write without blocking.
+.SH EXAMPLE
+.nf
+CURL *easy_handle;
+CURLM *multi_handle;
+
+/* add the individual easy handle */
+curl_multi_add_handle(multi_handle, easy_handle);
+
+do {
+  CURLMcode mc;
+  int numfds;
+
+  mc = curl_multi_perform(multi_handle, &still_running);
+
+  if(mc == CURLM_OK) {
+    /* wait for activity or timeout */
+    mc = curl_multi_poll(multi_handle, NULL, 0, 1000, &numfds);
+  }
+
+  if(mc != CURLM_OK) {
+    fprintf(stderr, "curl_multi failed, code %d.\\n", mc);
+    break;
+  }
+
+} while(still_running);
+
+curl_multi_remove_handle(multi_handle, easy_handle);
+.fi
+.SH RETURN VALUE
+CURLMcode type, general libcurl multi interface error code. See
+\fIlibcurl-errors(3)\fP
+.SH AVAILABILITY
+This function was added in libcurl 7.66.0.
+.SH "SEE ALSO"
+.BR curl_multi_fdset "(3), " curl_multi_perform "(3), " curl_multi_wait "(3)"