connections: introduce http/3 happy eyeballs

New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
  the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
  succeed after 100ms (subject to change), a parallel attempt
  is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
  as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
  expires, the HTTPS-CONNECT filter checks if the QUIC filter
  has received any data from the server. If not, it will start
  the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
  for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
  implementation into the udp socket cfilter. Because unconnected
  UDP sockets are weird. For example they error when adding to a
  pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
  and closes immediately (so the local conn goes into DRAIN phase), the
  connection is torn down and a another attempt is made after a short grace
  period.
  This is the behaviour observed with nghttpx when we tell it to  shut
  down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
  Invoke:
    python3 tests/tests-httpd/scorecard.py --help
  for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
  report connection statistics. This is triggered when the connection
  has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
  a timestamp of when it happend. This allows for updating timers
  "later", e.g. a connect statistic after full connectivity has been
  reached.
- in case of HTTP eyeballing, the previous changes will update
  statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
  by a filter chain.
  Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
  for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
  checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.

Closes #10349

1 files changed, 28 insertions, 65 deletions

diff --git a/lib/http.c b/lib/http.c
index 3c1711ab3..4d4f4fe87 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -219,38 +219,6 @@ const struct Curl_handler Curl_handler_wss = {
 
 #endif
 
-static CURLcode h3_setup_conn(struct Curl_easy *data,
-                              struct connectdata *conn)
-{
-#ifdef ENABLE_QUIC
-  /* We want HTTP/3 directly, setup the filter chain ourself,
-   * overriding the default behaviour. */
-  DEBUGASSERT(conn->transport == TRNSPRT_QUIC);
-
-  if(!(conn->handler->flags & PROTOPT_SSL)) {
-    failf(data, "HTTP/3 requested for non-HTTPS URL");
-    return CURLE_URL_MALFORMAT;
-  }
-#ifndef CURL_DISABLE_PROXY
-  if(conn->bits.socksproxy) {
-    failf(data, "HTTP/3 is not supported over a SOCKS proxy");
-    return CURLE_URL_MALFORMAT;
-  }
-  if(conn->bits.httpproxy && conn->bits.tunnel_proxy) {
-    failf(data, "HTTP/3 is not supported over a HTTP proxy");
-    return CURLE_URL_MALFORMAT;
-  }
-#endif
-
-  return CURLE_OK;
-#else /* ENABLE_QUIC */
-  (void)conn;
-  (void)data;
-  DEBUGF(infof(data, "QUIC is not supported in this build"));
-  return CURLE_NOT_BUILT_IN;
-#endif /* !ENABLE_QUIC */
-}
-
 static CURLcode http_setup_conn(struct Curl_easy *data,
                                 struct connectdata *conn)
 {
@@ -266,13 +234,16 @@ static CURLcode http_setup_conn(struct Curl_easy *data,
   Curl_mime_initpart(&http->form);
   data->req.p.http = http;
 
-  if(data->state.httpwant == CURL_HTTP_VERSION_3) {
+  if((data->state.httpwant == CURL_HTTP_VERSION_3)
+     || (data->state.httpwant == CURL_HTTP_VERSION_3ONLY)) {
+    CURLcode result = Curl_conn_may_http3(data, conn);
+    if(result)
+      return result;
+
+     /* TODO: HTTP lower version eyeballing */
     conn->transport = TRNSPRT_QUIC;
   }
 
-  if(conn->transport == TRNSPRT_QUIC) {
-    return h3_setup_conn(data, conn);
-  }
   return CURLE_OK;
 }
 
@@ -1320,7 +1291,7 @@ CURLcode Curl_buffer_send(struct dynbuf *in,
 
   DEBUGASSERT(socketindex <= SECONDARYSOCKET);
 
-  sockfd = conn->sock[socketindex];
+  sockfd = Curl_conn_get_socket(data, socketindex);
 
   /* The looping below is required since we use non-blocking sockets, but due
      to the circumstances we will just loop and try again and again etc */
@@ -1571,8 +1542,8 @@ static int http_getsock_do(struct Curl_easy *data,
                            curl_socket_t *socks)
 {
   /* write mode */
-  (void)data;
-  socks[0] = conn->sock[FIRSTSOCKET];
+  (void)conn;
+  socks[0] = Curl_conn_get_socket(data, FIRSTSOCKET);
   return GETSOCK_WRITESOCK(0);
 }
 
@@ -3008,33 +2979,25 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
      the rest of the request in the PERFORM phase. */
   *done = TRUE;
 
-  if(Curl_conn_is_http3(data, conn, FIRSTSOCKET)
-     || Curl_conn_is_http2(data, conn, FIRSTSOCKET)
-     || conn->httpversion == 20 /* like to get rid of this */) {
-    /* all fine, we are set */
-  }
-  else { /* undecided */
-    switch(conn->alpn) {
-    case CURL_HTTP_VERSION_2:
-      result = Curl_http2_switch(data, conn, FIRSTSOCKET, NULL, 0);
+  switch(conn->alpn) {
+  case CURL_HTTP_VERSION_3:
+    DEBUGASSERT(Curl_conn_is_http3(data, conn, FIRSTSOCKET));
+    break;
+  case CURL_HTTP_VERSION_2:
+    DEBUGASSERT(Curl_conn_is_http2(data, conn, FIRSTSOCKET));
+    break;
+  case CURL_HTTP_VERSION_1_1:
+    /* continue with HTTP/1.1 when explicitly requested */
+    break;
+  default:
+    /* Check if user wants to use HTTP/2 with clear TCP */
+    if(Curl_http2_may_switch(data, conn, FIRSTSOCKET)) {
+      DEBUGF(infof(data, "HTTP/2 over clean TCP"));
+      result = Curl_http2_switch(data, conn, FIRSTSOCKET);
       if(result)
         return result;
-      break;
-
-    case CURL_HTTP_VERSION_1_1:
-      /* continue with HTTP/1.1 when explicitly requested */
-      break;
-
-    default:
-      /* Check if user wants to use HTTP/2 with clear TCP */
-      if(Curl_http2_may_switch(data, conn, FIRSTSOCKET)) {
-        DEBUGF(infof(data, "HTTP/2 over clean TCP"));
-        result = Curl_http2_switch(data, conn, FIRSTSOCKET, NULL, 0);
-        if(result)
-          return result;
-      }
-      break;
     }
+    break;
   }
 
   http = data->req.p.http;
@@ -3936,8 +3899,8 @@ CURLcode Curl_http_readwrite_headers(struct Curl_easy *data,
 
             /* switch to http2 now. The bytes after response headers
                are also processed here, otherwise they are lost. */
-            result = Curl_http2_switch(data, conn, FIRSTSOCKET,
-                                       k->str, *nread);
+            result = Curl_http2_upgrade(data, conn, FIRSTSOCKET,
+                                        k->str, *nread);
             if(result)
               return result;
             *nread = 0;