| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
| |
Replace 'fd > 0' with 'fd >= 0' to avoid treating a possible file
descriptor as an error code.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
|
|
|
|
|
|
|
|
| |
debuginfod_find_section may attempt to download both the debuginfo
and executable matching the given build-id. If neither of these
files can be found, update rc to ensure that we always return an
accurate error code in this case.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
|
|
|
|
|
|
| |
The extract_section function in debuginfod-client.c is an internal
function and should not be exported. Mark it as static.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
| |
Return early from path_escape when '\0' is seen in order to prevent
an out-of-bounds write to the dest buffer.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add new function debuginfod_find_section which queries debuginfod
servers for the raw binary contents of the specified ELF/DWARF section
in a file matching the given build-id.
Extend the server webapi to support section queries. Section query
URLS have the following format: /buildid/BUILDID/section/SECTION
The server will attempt to extract the section from a debuginfo file
matching the given build-id. If the debuginfo file cannot be found
or the section has type SHT_NOBITS, the server will attempt to extract
the section from the executable file matching the build-id.
If the server is built without section query support, the client will
attempt to download the debuginfo matching the build-id and extract the
section. If the debuginfo file cannot be found or the section has type
SHT_NOBITS, the server will attempt to download the executable file
matching the build-id and extract the section.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
|
|
|
|
|
|
| |
Generalize lookup to "file" vs "glob" modes, propagate through API.
Add server side timeout for metadata query ops.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
| |
Debuginfod servers can be queried by source path and will return JSON
arrays of all the known metadata.
Signed-off-by: Ryan Goldberg <rgoldber@redhat.com>
|
|
|
|
|
|
|
|
| |
This DEBUGINFOD_HEADERS_FILE environment variable names a file to supply
HTTP headers to outgoing requests. Notably, this allows for
Authorization headers to be added from a file under OS access control.
Signed-off-by: Daniel Thornburgh <dthorn@google.com>
|
|
|
|
|
| |
Switch a couple error codes from positive to negative so they aren't
interpreted as file descriptors by the caller.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
On 32bit system compiled with -D_TIME_BITS=64, following tests failed:
FAIL: run-debuginfod-archive-test.sh
FAIL: run-debuginfod-archive-groom.sh
FAIL: run-debuginfod-archive-rename.sh
FAIL: run-debuginfod-federation-sqlite.sh
FAIL: run-debuginfod-federation-metrics.sh
FAIL: run-debuginfod-federation-link.sh
due to tests/debuginfod-subr.sh:archive_test failing on
"test $fileame -ot `pwd`", with LHS file having a bogus timestamp.
LHS is generated by debuginfo-find as a cache file.
I'm not sure the fix is the correct way, but
since -D_TIME_BITS=64 mandates -D_FILE_OFFSET_BITS=64 in glibc,
this does work on glibc environment.
Signed-off-by: Taketo Kabe <kabe@sra-tohoku.co.jp>
|
|
|
|
|
|
|
| |
Mark noted we need to ditch the last client ->winning_headers along
with the last ->url.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
| |
Signed-off-by: Martin Liska <mliska@suse.cz>
|
|
|
|
|
|
|
|
| |
Embrace case-independent headers, more fully document, handle HTTP \r.
In addition to test case, hand-tested against fedora debuginfod
instances, running federated servers under valgrind.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
|
| |
Debuginfod and debuginfod clients are now equipped to send
and receive http headers prefixed with X-DEBUGINFOD and
print them in verbose mode for more context
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Call utime on interval_path file as soon as the thread is committed to
cleanup the cache files. This will prevent other threads trying to
also commit to cleaning the cache files. Having multiple threads try
to clean the cache simultaniously doesn't improve cleanup speed
because the threads will try to delete the files in the same order.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
| |
debuginfod_init_cache would create all config files if they didn't
exist yet. It always made two stat calls. Then debuginfod_clean_cache
would call debuginfod_config_cache which did the same checks and
created any missing config files. Just make sure the cache_path
directory exists and remove debuginfod_init_cache before calling
debuginfod_clean_cache.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
| |
If the condig file which value was requested from
debuginfod_config_cache didn't exist yet, stat would fail and no valid
struct stat would be returned even when the file was correctly
created. Fix this by always using O_CREAT to open the file, and reuse
that file descriptor to call fstat and for either writing the default
value or reading the config file value.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
| |
curl_easy_setup can fail for various reasons. Add a curl_easy_setopt_ck
macro to check all curl_easy_setopt calls and provides a human readable
error message in verbose mode.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
| |
This was the only place in debuginfod-client.c where we didn't check
the result of curl_easy_getinfo. Just check it to make things consistent.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
| |
Correct a nasty fd leak and a few less nasty leaks in the debuginfod
client code. The nasty one impacts long-lived apps such as debuginfod
servers.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In some cases the content-length header may not be available in order
to pass to a progressfn. If content-length isn't available then attempt
to get the size of the download from the debuginfod-size header instead.
It should be mentioned that if a compressed file (ex. gzip) is being
transferred, the actual transfer length will be less than debuginfod-size.
In this case debuginfod-size is a best-guess upper bound on the size of
the transfer.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We used to go out of our way to initialize libcurl early before any other
thread/code was running. But this meant that we might pay startup cost,
which under FIPS is significant, even for code that never uses libdebuginfod
or TLS libcurl connections. Although curl_global_init itself isn't thread-safe
we can use pthread_once to make sure we don't race against ourselves. This
still means we might race against any application code that might use
libcurl. But we can assume they will have called curl_global_init before
calling dwfl_begin or debuginfod_begin.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
000-permission files currently used for negative caching can cause
permission problems for some backup software and disk usage checkers.
Fix this by using empty files for negative caching instead.
Also use each empty file's mtime to determine the time since
last download attempt instead of the cache_miss_s file's mtime.
https://sourceware.org/bugzilla/show_bug.cgi?id=29022
Tested-by: Milian Wolff <mail@milianw.de>
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Use a single MHD_USE_DUAL_STACK mhd daemon. This way, the thread
connection pool is not doubled, saving memory and better matching user
expectations. A slight tweak to logging is required to pull IPv4
remote addresses back out, and also to allow IPv6 ::-laden address
forwarding through federation links.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
time_t is platform dependent and some of architectures e.g.
x32, riscv32, arc use 64bit time_t even while they are 32bit
architectures, therefore directly using integer printf formats will not
work portably.
Use a plain long everywhere as the intervals are small enough
that it will not be problematic.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
|
|
|
|
|
|
|
|
| |
gcc address sanitizer detected a leak of the debuginfod_client
winning_headers when the handle was reused. Make sure to free and
reset the winning_headers field before reuse.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
| |
In a couple of places we might leak some memory when we encounter
an error. tmp_url might leak if realloc failed. escaped_string might
leak when setting up the data handle fails and we don't use it.
And one of the goto out1 should have been goto out2 to make sure
we release all allocated resources on exit (also updated a wrong
comment about that).
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Negative cache (000-permission) files were incorrectly treated as
valid cached files for the root user, because root can open even
000-perm files without -EACCES. Corrected this checking sequence.
Fixed the debuginfod testsuite to run to completion as root or
as an ordinary user, correcting corresponding permission checks:
stat -c %A $FILE
is right and
[ -w $FILE] [ -r $FILE ]
were wrong.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Make sure we don't use any of the more experimental protocols
libcurl might support. URLs can be redirected and we might want
to follow http -> https, but not e.g. gopher or pop3.
Suggested-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
While checking PR27277 on some buildbots, greping would fail in
run-debuginfod-response-headers.sh. This was because querying the
debuginfod server occurs after checking if the responseh headers had
arrived, leaving the possibility to leave the querying loop before
outputting the headers which caused the grep failure. Querying now
occurs before checking if response headers have arrived, so that they
will certainly be printed and grep will find them.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allow users, with enough verbosity, to print the HTTP response headers
upon retrieving a file. These files may include several custome http
response headers such as X-DEBUGINFOD-FILE, X-DEBUGINFOD-SIZE, and
X-DEBUGINFOD-ARCHIVE. These headers are added from the daemon, in
debuginfod.cxx.
E.g output:
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 4095072
Cache-Control: public
Last-Modified: Thu, 09 Sep 2021 19:06:40 GMT
X-FILE: debuginfod
X-FILE-SIZE: 4095072
Content-Type: application/octet-stream
Date: Fri, 10 Sep 2021 16:38:06 GMT
https://sourceware.org/bugzilla/show_bug.cgi?id=27277
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, urls containing '/', so most urls, would escape '/' to %2F,
which is undesirable for use in other libraries which may escape
differently. This patch escapes the '/' and replaces all of them
ensuring there are no %2Fs sent.
Some inefficiencies within the code were fixed, such as changing constant
operations of a while loop within a for loop to a while loop outside of
a for loop. Also strlen is no longer used within the loop, simplifying
the interior operations to mere arithmetic.
https://sourceware.org/bugzilla/show_bug.cgi?id=28034
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
|
|
|
|
|
|
| |
Return values of functions returning "void *", e.g. calloc, malloc,
realloc, xcalloc, xmalloc, and xrealloc, do not need explicit casts.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
DEBUGINFOD_TIMEOUT is a good way to catch servers that are too slow to
*start* transmitting a file. But we have no way of limiting total
download time or space. A user might prefer to have his debugger fetch
only quick & small files, and make do without the bigger ones. Some
transitive dependencies of e.g. gnome programs are huge: 3GB of LLVM
debuginfo, 1GB of webkitgtk, etc. etc.
DEBUGINFOD_MAXSIZE and DEBUGINFOD_MAXTIME were added to dictate the
max download size and time of a debuginfod client. DEBUGINFOD_MAXSIZE
is handled server-side and is sent using the http header:
X-DEBUGINFOD-MAXSIZE. The client side then checks to ensure this maxsize
has been respected.
https://sourceware.org/bugzilla/show_bug.cgi?id=27982
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Gazing at server logs, one sees a minority of clients who appear to have
duplicate query traffic coming in: the same URL, milliseconds apart.
Chances are the user accidentally doubled her $DEBUGINFOD_URLS somehow,
and the client library is dutifully asking the servers TWICE. Bug #27863
reduces the pain on the servers' CPU, but dupe network traffic is still
being paid. We should reject sending outright duplicate concurrent
traffic.
The urls are now simply removed upon finding a duplicate after url
construction.
https://sourceware.org/bugzilla/show_bug.cgi?id=27983
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When requesting some source files, some URL-inconvenient chars
sometimes pop up. Example from f33 libstdc++:
/buildid/44d8485cb75512c2ca5c8f70afbd475cae30af4f/source/usr/src/debug/
gcc-10.3.1-1.fc33.x86_64/obj-x86_64-redhat-linux/x86_64-redhat-linux/
libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/
condition_variable.cc
As this URL is passed into debuginfod's handler_cb, it appears that the
+ signs are helpfully unescaped to spaces by libmicrohttpd, which
'course breaks everything.
In order to ensure the server properly parses urls such as this one,
%-escape characters on the client side so that the correct url
is preserved and properly processed on the server side.
https://sourceware.org/bugzilla/show_bug.cgi?id=28034
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In debuginfod-client.c (debuginfod_query_server),insert a
goto statement for jumping back to the beginning of curl
handles set up if query fails and a non ENOENT error is returned.
Also introduced DEBUGINFOD_RETRY_LIMIT_ENV_VAR and default
DEBUGINFOD_RETRY_LIMIT(which is 2).
Correponding test has been added to tests/run-debuginfod-find.sh
Signed-off-by: Alice Zhang <alizhang@redhat.com>
|
|
|
|
|
|
|
| |
When the calloc call in debuginfod_begin fails we should skip all
initialization of the client handle.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
PR27701 implemented curl handle reuse in debuginfod_client objects,
but with an unexpected bug. Server responses returning an error
"latched" because the curl_easy handles for error cases weren't all
systematically removed from the curl multi handle. This prevented
their proper re-addition the next time.
This version of the code simplfies matters by making only the curl
curl_multi handle long-lived. This turns out to be enough, because it
can maintain a pool of long-lived http/https connections and related
data, and lend them out to short-lived curl_easy handles. This mode
handles errors or hung downloads even better, because the easy handles
don't undergo complex state transitions between reuse.
A new test case confirms this correction via the federating debuginfod
instance (cleaning caches between subtests to make sure http* is being
used and reused).
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
| |
New function in system.h that returns true if a string has a given
prefix, false otherwise. Use it in place of strncmp.
Signed-off-by: Martin Liška <mliska@suse.cz>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add debuginfod_config_cache for reading and writing to cache
configuration files, make use of the function within
debuginfod_clean_cache and debuginfod_query_server.
In debuginfod_query_server, create 000-permission file on failed
queries. Before querying each BUILDID, if corresponding 000 file
detected, compare its stat mtime with parameter from
.cache/cache_miss_s. If mtime is fresher, then return ENOENT and
exit; otherwise unlink the 000 file and proceed to a new query.
tests: add test in run-debuginfod-find.sh
test if the 000 file is created on failed query; if querying the
same failed BUILDID, whether the query should proceed without
going through server; set the cache_miss_s to 0 and query the same
buildid, and this time should go through the server.
Signed-off-by: Alice Zhang <alizhang@redhat.com>
|
|
|
|
|
|
| |
Files in the download cache should be read-only.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
|
|
| |
With PR25365, we accidentally lost the ability to rmdir client-cache
directories corresponding to buildids. Bring this back, with some
attention to a possible race between a client doing cleanup and
another client doing lookups at the same time.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Client objects now carry long-lived curl handles for outgoing
connections. This makes it more efficient for multiple sequential
queries, because the TCP connections and/or TLS state info are kept
around awhile, avoiding O(100ms) setup latencies. debuginfod is
adjusted to take advantage of this for federation. Other clients
should gradually do this too, perhaps including elfutils itself (in
the libdwfl->debuginfod_client hooks).
A large gdb session with 117 debuginfo downloads was observed to run
twice as fast (45s vs. 1m30s wall-clock time), just in nuking this
extra setup latency. This was tested via a debuginfod intermediary:
it should be even faster once gdb reuses its own debuginfod_client.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Clang warns about this:
../../debuginfod/debuginfod-client.c:899:28: error: implicit conversion from 'long' to 'double' changes value from 9223372036854775807 to 9223372036854775808 [-Werror,-Wimplicit-int-float-conversion]
pa = (dl > LONG_MAX ? LONG_MAX : (long)dl);
~ ^~~~~~~~
/usr/lib64/clang/10.0.1/include/limits.h:47:19: note: expanded from macro 'LONG_MAX'
^~~~~~~~~~~~
<built-in>:38:22: note: expanded from here
^~~~~~~~~~~~~~~~~~~~
Modified for jakub's observation about LONG_MAX overflow.
Signed-off-by: Timm Bäder <tbaeder@redhat.com>
|
|
|
|
|
|
| |
CURLINFO_CURLINFO_CONTENT_LENGTH_DOWNLOAD_T is one CURLINFO_ too much.
Signed-off-by: Timm Bäder <tbaeder@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Initalize -> Initialize
Unsucessful -> Unsuccessful
expession -> expression
incompatiblity -> incompatibility
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
|
|
|
|
| |
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduce a new function debuginfod_set_verbose_fd which will produce
verbose output on a given file descriptor (STDERR_FILENO if the
environment variable DEBUGINFOD_VERBOSE is set) showing how the search
for a particular client query is going.
Example output:
debuginfod_find_debuginfo 1234567890
server urls "https://debuginfod.elfutils.org/ http://dbgd.usersys.com:3632/"
checking build-id
checking cache dir /home/mark/.cache/debuginfod_client
using timeout 90
init server 0 https://debuginfod.elfutils.org/
url 0 https://debuginfod.elfutils.org/buildid/1234567890/debuginfo
init server 1 http://dbgd.usersys.com:3632/
url 1 http://dbgd.usersys.com:3632/buildid/1234567890/debuginfo
query 2 urls in parallel
server response HTTP response code said error
url 0 The requested URL returned error: 404 Not Found
server response HTTP response code said error
url 1 The requested URL returned error: 404 Not Found
not found No such file or directory (err=-2)
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Use defined constants for permission values. Also add fallback
definitions for them in system.h, to allow for compatibility with
systems that don't provide these macros.
Include system.h in all tests/ files that required it.
Signed-off-by: Érico Rolim <erico.erc@gmail.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
|