summaryrefslogtreecommitdiff
path: root/lib/transfer.c
Commit message (Collapse)AuthorAgeFilesLines
* transfer: refuse POSTFIELDS + RESUME_FROM comboDaniel Stenberg2023-05-081-0/+6
| | | | | | | | | | | | | The code assumes that such a resume is wanting to continue an upload using the read callback, and since POSTFIELDS is done without callback libcurl will just misbehave. This combo will make the transfer fail with CURLE_BAD_FUNCTION_ARGUMENT with an explanation in the error message. Reported-by: Smackd0wn on github Fixes #11081 Closes #11083
* h2/h3: replace `state.drain` counter with `state.dselect_bits`Stefan Eissing2023-04-251-32/+28
| | | | | | | | | | | | | | | | | | | | | - `drain` was used by http/2 and http/3 implementations to indicate that the transfer requires send/recv independant from its socket poll state. Intended as a counter, it was used as bool flag only. - a similar mechanism exists on `connectdata->cselect_bits` where specific protocols can indicate something similar, only for the whole connection. - `cselect_bits` are cleard in transfer.c on use and, importantly, also set when the transfer loop expended its `maxloops` tries. `drain` was not cleared by transfer and the http2/3 implementations had to take care of that. - `dselect_bits` is cleared *and* set by the transfer loop. http2/3 does no longer clear it, only set when new events happen. This change unifies the handling of socket poll overrides, extending `cselect_bits` by a easy handle specific value and a common treatment in transfers. Closes #11005
* lib: unify the upload/method handlingDaniel Stenberg2023-04-251-2/+2
| | | | | | | | By making sure we set state.upload based on the set.method value and not independently as set.upload, we reduce confusion and mixup risks, both internally and externally. Closes #11017
* transfer: skip extra assignDaniel Stenberg2023-04-131-1/+0
| | | | | | | | The 'result' variable already contains CURLE_OK at this point, no use in setting it again. Pointed out by PVS. Ref: #10929 Closes #10944
* multi: remove PENDING + MSGSENT handles from the main linked listDaniel Stenberg2023-03-261-1/+6
| | | | | | | | | | | | | | | | | | | | | | As they are not driving transfers or any socket activity, the main loop does not need to iterate over these handles. A performance improvement. They are instead only held in their own separate lists. 'data->multi' is kept a pointer to the multi handle as long as the easy handle is actually part of it even when the handle is moved to the pending/msgsent lists. It needs to know which multi handle it belongs to, if for example curl_easy_cleanup() is called before the handle is removed from the multi handle. Alll 'data->multi' pointers of handles still part of the multi handle gets cleared by curl_multi_cleanup() which "orphans" all previously attached easy handles. This is take 2. The first version was reverted for the 8.0.1 release. Assisted-by: Stefan Eissing Closes #10801
* http2: Use KEEP_SEND_HOLD for flow control in HTTP/2Stefan Eissing2023-03-141-2/+1
| | | | | | | | | | | - use the defined, but so far not used, KEEP_SEND_HOLD bit for flow control based suspend of sending in transfers. Prior to this change KEEP_SEND_PAUSE bit was used instead, but that can interfere with pausing streams from the user side via curl_easy_pause. Fixes https://github.com/curl/curl/issues/10751 Closes https://github.com/curl/curl/pull/10753
* ftp: allocate the wildcard struct on demandDaniel Stenberg2023-03-031-1/+7
| | | | | | | | | | The feature is rarely used so this frees up data for the vast majority of easy handles that don't use it. Rename "protdata" to "ftpwc" since it is always an FTP wildcard struct pointer. Made the state struct field an unsigned char to save space. Closes #10639
* transfer: limit Windows SO_SNDBUF updates to once a secondJay Satiro2023-03-011-1/+9
| | | | | | | | | | | | | | | | - Change readwrite_upload() to call win_update_buffer_size() no more than once a second to update SO_SNDBUF (send buffer limit). Prior to this change during an upload readwrite_upload() could call win_update_buffer_size() anywhere from hundreds of times per second to an extreme test case of 100k per second (which is likely due to a bug, see #10618). In the latter case WPA profiler showed win_update_buffer_size was the highest capture count in readwrite_upload. In any case the calls were excessive and unnecessary. Ref: https://github.com/curl/curl/pull/2762 Closes https://github.com/curl/curl/pull/10611
* urldata: remove `now` from struct SingleRequest - not neededStefan Eissing2023-02-171-6/+7
| | | | Closes #10549
* HTTP/[23]: continue upload when state.drain is setStefan Eissing2023-02-091-0/+2
| | | | | | | | | | | | | | | - as reported in #10433, HTTP/2 uploads may stall when a response is received before the upload is done. This happens when the data->state.drain is set for such a transfer, as the special handling in transfer.c from then on only cared about downloads. - add continuation of uploads, if applicable, in this case. - add pytest case test_07_12_upload_seq_large to reproduce this scenario (although, current nghttp2 implementation is using drain less often) Reported-by: Lucas Pardue Fixes #10433 Closes #10443
* copyright: update all copyright lines and remove year rangesDaniel Stenberg2023-01-031-1/+1
| | | | | | | | | | | | - they are mostly pointless in all major jurisdictions - many big corporations and projects already don't use them - saves us from pointless churn - git keeps history for us - the year range is kept in COPYING checksrc is updated to allow non-year using copyright statements Closes #10205
* lib: connect/h2/h3 refactorStefan Eissing2022-12-301-60/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactoring of connection setup and happy eyeballing. Move nghttp2. ngtcp2, quiche and msh3 into connection filters. - eyeballing cfilter that uses sub-filters for performing parallel connects - socket cfilter for all transport types, including QUIC - QUIC implementations in cfilter, can now participate in eyeballing - connection setup is more dynamic in order to adapt to what filter did really connect. Relevant to see if a SSL filter needs to be added or if SSL has already been provided - HTTP/3 test cases similar to HTTP/2 - multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche - Fix for data attach/detach in VTLS filters that could lead to crashes during parallel transfers. - Eliminating setup() methods in cfilters, no longer needed. - Improving Curl_conn_is_alive() to replace Curl_connalive() and integrated ssl alive checks into cfilter. - Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update connection into and persist it at the easy handle. - Several more cfilter related cleanups and moves: - stream_weigth and dependency info is now wrapped in struct Curl_data_priority - Curl_data_priority members depend is available in HTTP2|HTTP3 - Curl_data_priority members depend on NGHTTP2 support - handling init/reset/cleanup of priority part of url.c - data->state.priority same struct, but shallow copy for compares only - PROTOPT_STREAM has been removed - Curl_conn_is_mulitplex() now available to check on capability - Adding query method to connection filters. - ngtcp2+quiche: implementing query for max concurrent transfers. - Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event. - setting keepalive timestamp on connect - DATA_SETUP is called after the connection has been completely setup (but may not connected yet) to allow filters to initialize data members they use. - there is no socket to be had with msh3, it is unclear how select shall work - manual test via "curl --http3 https://curl.se" fail with "empty reply from server". - Various socket/conn related cleanups: - Curl_socket is now Curl_socket_open and in cf-socket.c - Curl_closesocket is now Curl_socket_close and in cf-socket.c - Curl_ssl_use has been replaced with Cur_conn_is_ssl - Curl_conn_tcp_accepted_set has been split into Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set with a clearer purpose Closes #10141
* transfer: break the read loop when RECV is clearedDaniel Stenberg2022-12-301-2/+2
| | | | | | | | | | When the RECV bit is cleared because the response reading for this transfer is complete, the read loop should be stopped. data_pending() can otherwise still return TRUE and another read would be attempted. Reported-by: Hide Ishikawa Fixes #10172 Closes #10174
* share: add sharing of HSTS cache among handlesDaniel Stenberg2022-12-271-0/+3
| | | | Closes #10138
* urldata: move the cookefilelist to the 'set' structDaniel Stenberg2022-12-221-4/+2
| | | | | | | | | The cookiefile entries are set into the handle and should remain set for the lifetime of the handle so that duplicating it also duplicates the list. Therefore, the struct field is moved from 'state' to 'set'. Fixes #10133 Closes #10134
* lib: rewind BEFORE request instead of AFTER previousDaniel Stenberg2022-11-251-96/+4
| | | | | | | | | | | | This makes a big difference for cases when the rewind is not actually necessary to perofm (for example HTTP response code 301 converts to GET) and therefore the rewind can be avoided. In particular for situations when that rewind fails, for example when reading from a pipe or similar. Reported-by: Ali Utku Selen Fixes #9735 Closes #9958
* vtls: localization of state data in filtersStefan Eissing2022-11-221-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - almost all backend calls pass the Curl_cfilter intance instead of connectdata+sockindex - ssl_connect_data is remove from struct connectdata and made internal to vtls - ssl_connect_data is allocated in the added filter, kept at cf->ctx - added function to let a ssl filter access its ssl_primary_config and ssl_config_data this selects the propert subfields in conn and data, for filters added as plain or proxy - adjusted all backends to use the changed api - adjusted all backends to access config data via the exposed functions, no longer using conn or data directly cfilter renames for clear purpose: - methods `Curl_conn_*(data, conn, sockindex)` work on the complete filter chain at `sockindex` and connection `conn`. - methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter instance. - methods `Curl_conn_cf()` work on/with filter instances at a connection. - rebased and resolved some naming conflicts - hostname validation (und session lookup) on SECONDARY use the same name as on FIRST (again). new debug macros and removing connectdata from function signatures where not needed. adapting schannel for new Curl_read_plain paramter. Closes #9919
* http: mark it 'this_is_a_follow' in the Location: logicDaniel Stenberg2022-11-121-4/+2
| | | | | | | | | To make regular auth "reloads" to not count as redirects. Verified by test 3101 Fixes #9885 Closes #9887
* lib: remove bad set.opt_no_body assignmentsDaniel Stenberg2022-11-111-7/+7
| | | | | | | | | | This struct field MUST remain what the application set it to, so that handle reuse and handle duplication work. Instead, the request state bit 'no_body' is introduced for code flows that need to change this in run-time. Closes #9888
* lib: connection filters (cfilter) addition to curl:Stefan Eissing2022-11-111-28/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | - general construct/destroy in connectdata - default implementations of callback functions - connect: cfilters for connect and accept - socks: cfilter for socks proxying - http_proxy: cfilter for http proxy tunneling - vtls: cfilters for primary and proxy ssl - change in general handling of data/conn - Curl_cfilter_setup() sets up filter chain based on data settings, if none are installed by the protocol handler setup - Curl_cfilter_connect() boot straps filters into `connected` status, used by handlers and multi to reach further stages - Curl_cfilter_is_connected() to check if a conn is connected, e.g. all filters have done their work - Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE indicators for multi select to work - Curl_cfilter_data_pending() asks filters if the have incoming data pending for recv - Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks installed in conn->recv/conn->send for io handling - Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters and addition/removal of a `data` from their connection - adding vtl functions to prevent use of Curl_ssl globals directly in other parts of the code. Reviewed-by: Daniel Stenberg Closes #9855
* misc: typo and grammar fixesAyesh Karunaratne2022-10-271-2/+2
| | | | | | | | | | - Replace `Github` with `GitHub`. - Replace `windows` with `Windows` - Replace `advice` with `advise` where a verb is used. - A few fixes on removing repeated words. - Replace `a HTTP` with `an HTTP` Closes #9802
* setopt: use the handler table for protocol name to number conversionsPatrick Monnerat2022-09-161-1/+1
| | | | | | | | | | | | | | This also returns error CURLE_UNSUPPORTED_PROTOCOL rather than CURLE_BAD_FUNCTION_ARGUMENT when a listed protocol name is not found. A new schemelen parameter is added to Curl_builtin_scheme() to support this extended use. Note that disabled protocols are not recognized anymore. Tests adapted accordingly. Closes #9472
* urlapi: detect scheme better when not guessingDaniel Stenberg2022-09-151-2/+5
| | | | | | | | | | | | | | | | When the parser is not allowed to guess scheme, it should consider the word ending at the first colon to be the scheme, independently of number of slashes. The parser now checks that the scheme is known before it counts slashes, to improve the error messge for URLs with unknown schemes and maybe no slashes. When following redirects, no scheme guessing is allowed and therefore this change effectively prevents redirects to unknown schemes such as "data". Fixes #9503
* headers: reset the requests counter at transfer startDaniel Stenberg2022-09-091-0/+1
| | | | | | | | | | | | | If not, reusing an easy handle to do a subsequent transfer would continue the counter from the previous invoke, which then would make use of the header API difficult/impossible as the request counter mismatched. Add libtest 1947 to verify. Reported-by: Andrew Lambert Fixes #9424 Closes #9447
* ngtcp2: fix stall or busy loop on STOP_SENDING with upload dataTatsuhiro Tsujikawa2022-07-101-1/+18
| | | | | Fixes #9122 Closes #9123
* transfer: upload performance; avoid tiny sendGlenn Strauss2022-06-201-7/+26
| | | | | | | | | | | | | Append to the upload buffer when only small amount remains in buffer rather than performing a separate tiny send to empty buffer. Avoid degenerative upload behavior which might cause curl to send mostly 1-byte DATA frames after exhausing the h2 send window size Related discussion: https://github.com/nghttp2/nghttp2/issues/1722 Signed-off-by: Glenn Strauss <gstrauss@gluelogic.com> Closes #8965
* copyright: make repository REUSE compliantmax.mehl2022-06-131-0/+2
| | | | | | | | | | | Add licensing and copyright information for all files in this repository. This either happens in the file itself as a comment header or in the file `.reuse/dep5`. This commit also adds a Github workflow to check pull requests and adapts copyright.pl to the changes. Closes #8869
* transfer: maintain --path-as-is after redirectsDaniel Stenberg2022-06-081-1/+2
| | | | | | Reported-by: Marcus T Fixes #8974 Closes #8975
* misc: spelling improvementsFabian Keil2022-06-051-1/+1
| | | | Closes #8956
* easy/transfer: fix cookie-disabled buildDaniel Stenberg2022-06-021-1/+2
| | | | | | | Follow-up from 45de940cebf6a Reported-by: Marcel Raad Fixes #8953 Closes #8954
* lib: make more protocol specific struct fields #ifdefedDaniel Stenberg2022-06-011-12/+13
| | | | | | | ... so that they don't take up space if the protocols are disabled in the build. Closes #8944
* transfer: Fix potential NULL pointer dereferenceDaniel Gustafsson2022-05-161-2/+5
| | | | | | | | | Commit 0ef54abf5208 accidentally used the conn variable before the assertion for it being NULL. Fix by moving the assignment which use conn to after the assertion. Closes: #8857 Reviewed-by: Daniel Stenberg <daniel@haxx.se>
* quic: add Curl_quic_idleTatsuhiro Tsujikawa2022-05-161-0/+8
| | | | | | | | Add Curl_quic_idle which is called when no HTTP level read or write is performed. It is a good place to handle timer expiry for QUIC transport (.e.g, retransmission). Closes #8698
* transfer: redirects to other protocols or ports clear authDaniel Stenberg2022-04-251-1/+48
| | | | | | | | ... unless explicitly permitted. Bug: https://curl.se/docs/CVE-2022-27774.html Reported-by: Harry Sintonen Closes #8748
* lib: remove exclamation marksDaniel Stenberg2022-04-161-5/+5
| | | | | | ... from infof() and failf() calls. Make them less attention seeking. Closes #8713
* lib: #ifdef on USE_HTTP2 betterDaniel Stenberg2022-04-011-9/+8
| | | | | | ... as nghttp2 might not be the library that provides HTTP/2 support. Closes #8661
* header api: add curl_easy_header and curl_easy_nextheaderDaniel Stenberg2022-03-221-0/+4
| | | | | | Add test 1940 to 1946 to verify. Closes #8593
* misc: remove strlen for Curl_checkheaders + Curl_checkProxyheadersHenrikHolst2022-02-101-2/+2
| | | | Closes #8409
* lib: remove support for CURL_DOES_CONVERSIONSDaniel Stenberg2022-02-041-45/+1
| | | | | | TPF was the only user and support for that was dropped. Closes #8378
* urlapi: make Curl_is_absolute_url always use MAX_SCHEME_LENDaniel Stenberg2021-11-251-1/+1
| | | | | | | | | | | | Instad of having all callers pass in the maximum length, always use it. The passed in length is instead used only as the length of the target buffer for to storing the scheme name in, if used. Added the scheme max length restriction to the curl_url_set.3 man page. Follow-up to 45bcb2eaa78c79 Closes #8047
* hsts: CURLSTS_FAIL from hsts read callback should fail transferDaniel Stenberg2021-09-161-1/+1
| | | | | | | | | | ... and have CURLE_ABORTED_BY_CALLBACK returned. Extended test 1915 to verify. Reported-by: Jonathan Cardoso Fixes #7726 Closes #7729
* infof: remove newline from format strings, always append itDaniel Stenberg2021-07-071-25/+25
| | | | | | | | | | | | | | | | - the data needs to be "line-based" anyway since it's also passed to the debug callback/application - it makes infof() work like failf() and consistency is good - there's an assert that triggers on newlines in the format string - Also removes a few instances of "..." - Removes the code that would append "..." to the end of the data *iff* it was truncated in infof() Closes #7357
* curl_url_set: reject spaces in URLs w/o CURLU_ALLOW_SPACEDaniel Stenberg2021-06-151-1/+2
| | | | | | | | | | | | | They were never officially allowed and slipped in only due to sloppy parsing. Spaces (ascii 32) should be correctly encoded (to %20) before being part of a URL. The new flag bit CURLU_ALLOW_SPACE when a full URL is set, makes libcurl allow spaces. Updated test 1560 to verify. Closes #7073
* [Win32] Fix for USE_WATT32Gisle Vanem2021-06-041-1/+1
| | | My Watt-32 tcp/ip stack works on Windows but it does not have `WSAIoctl()`
* c-hyper: fix NTLM on closed connection tested with test159Daniel Stenberg2021-06-011-2/+9
| | | | Closes #7154
* data_pending: check only SECONDARY socket for FTP(S) transfersJoel Depooter2021-05-181-4/+5
| | | | | | | | | | | | | | | | Check the FIRST for all other protocols. This fixes a timeout in an ftps download. The server sends a TLS close_notify message in the same packet as the file data. The close_notify seems to not be handled in the schannel_recv function, so libcurl is not aware that the server has closed the connection. Thus libcurl ends up waiting for action on the socket until a timeout is reached. With the secondary socket check added to the data_pending function, the close_notify is properly handled, and the ftps transfer terminates as expected. Fixes #7068 Closes #7069
* http: fix the check for 'Authorization' with BearerDaniel Stenberg2021-05-021-0/+2
| | | | | | | The code would wrongly check for it using an additional colon. Reported-by: Blake Burkhart Closes #6988
* lib: fix 0-length Curl_client_write callsJacob Hoffman-Andrews2021-04-291-2/+2
| | | | Closes #6954
* transfer: clear 'referer' in declarationDaniel Stenberg2021-03-291-2/+2
| | | | | | | | | To silence (false positive) compiler warnings about it. Follow-up to 7214288898f5625 Reviewed-by: Marcel Raad Closes #6810
* transfer: strip credentials from the auto-referer header fieldViktor Szakats2021-03-281-2/+23
| | | | | | | | Added test 2081 to verify. CVE-2021-22876 Bug: https://curl.se/docs/CVE-2021-22876.html
View Lorry Controller Status