| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| ... | |
| |
|
|
|
|
|
| |
This was the only place in debuginfod-client.c where we didn't check
the result of curl_easy_getinfo. Just check it to make things consistent.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
| |
Also disable MHD_USE_THREAD_PER_CONNECTION when using MHD_USE_EPOLL.
https://sourceware.org/bugzilla/show_bug.cgi?id=29123
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
|
| |
On a systems that have ipv6 disabled debuginfod doesn't start up
anymore because libhttpd MHD_USE_DUAL_STACK only works if it can
open an ipv6 socket. If MHD_start_daemon with MHD_USE_DUAL_STACK
fails try again without that flag set.
https://sourceware.org/bugzilla/show_bug.cgi?id=29122
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
| |
Added default value to fdcache_prefetch_mds and fdcache_prefetch_fds.
Defaults to one half of corresponging fdcache's values.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
foo
|
| |
|
|
|
|
|
|
|
| |
Correct a nasty fd leak and a few less nasty leaks in the debuginfod
client code. The nasty one impacts long-lived apps such as debuginfod
servers.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
For archived files X-DEBUGINFOD-SIZE currently contains the size of the
archive instead of the size of the uncompressed file. Fix this.
Also add testcases to verify X-DEBUGINFOD-SIZE contains uncompressed
file sizes.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
In some cases the content-length header may not be available in order
to pass to a progressfn. If content-length isn't available then attempt
to get the size of the download from the debuginfod-size header instead.
It should be mentioned that if a compressed file (ex. gzip) is being
transferred, the actual transfer length will be less than debuginfod-size.
In this case debuginfod-size is a best-guess upper bound on the size of
the transfer.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
We used to go out of our way to initialize libcurl early before any other
thread/code was running. But this meant that we might pay startup cost,
which under FIPS is significant, even for code that never uses libdebuginfod
or TLS libcurl connections. Although curl_global_init itself isn't thread-safe
we can use pthread_once to make sure we don't race against ourselves. This
still means we might race against any application code that might use
libcurl. But we can assume they will have called curl_global_init before
calling dwfl_begin or debuginfod_begin.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This prevents the "Server reached connection limit. Closing inbound
connection." issue we have been seeing in the
run-debuginfod-webapi-concurrency.sh testcase. From the manual:
If the connection limit is reached, MHD’s behavior depends a bit
on other options. If MHD_USE_ITC was given, MHD will stop
accepting connections on the listen socket. This will cause the
operating system to queue connections (up to the listen() limit)
above the connection limit. Those connections will be held until
MHD is done processing at least one of the active connections. If
MHD_USE_ITC is not set, then MHD will continue to accept() and
immediately close() these connections.
https://sourceware.org/bugzilla/show_bug.cgi?id=28708
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
000-permission files currently used for negative caching can cause
permission problems for some backup software and disk usage checkers.
Fix this by using empty files for negative caching instead.
Also use each empty file's mtime to determine the time since
last download attempt instead of the cache_miss_s file's mtime.
https://sourceware.org/bugzilla/show_bug.cgi?id=29022
Tested-by: Milian Wolff <mail@milianw.de>
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
At startup debuginfod prints a message indicating the port which the
server is listening to. Prior to commit 4e4082be03 this message would
include "IPv4" and/or "IPv6"
[...] (48671/48671): started http server on IPv4 IPv6 port=8002
As of commit 4e4082be03 the IP versions have been removed from this
message. This change can cause issues in any applications that
parse the message for this information. Fix this by adding
"IPv4 IPv6" back to the message.
Signed-off-by: Aaron Merey <amerey@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Use a single MHD_USE_DUAL_STACK mhd daemon. This way, the thread
connection pool is not doubled, saving memory and better matching user
expectations. A slight tweak to logging is required to pull IPv4
remote addresses back out, and also to allow IPv6 ::-laden address
forwarding through federation links.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
Testing on s390x and other architectures indicates that this
configuration reduces thundering-herd wakeups and saturation of a
small number of threads. The run-debuginfod-webapi-concurrency.sh
test appears solid now.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
time_t is platform dependent and some of architectures e.g.
x32, riscv32, arc use 64bit time_t even while they are 32bit
architectures, therefore directly using integer printf formats will not
work portably.
Use a plain long everywhere as the intervals are small enough
that it will not be problematic.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Add an option -C, which activates libmicrohttpd's thread-pool mode for
handling incoming http connections. Add libmicrohttpd error-logging
callback function so as to receive indication of its internal errors,
and relay counts to our metrics. Some of these internal errors tipped
us off to a microhttpd bug that thread pooling works around. Document
in debuginfod.8 page. Hand-tested against "ulimit -u NNN" shells, and
with a less strenuous new test case.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Keep the logs just plain unformatted text.
This really is a workaround for an apparent bug with gcc 8.3
-fsanitizer=undefined on arm32, which complains about the
'right' formatter:
debuginfod.cxx:3472:12: runtime error: reference binding to
misaligned address 0x00561ec9 for type '<unknown>', which
requires 2 byte alignment
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
| |
gcc address sanitizer detected a dangling debuginfod_client handler
when debuginfod exits. Make sure to groom the debuginfod client pool
before exit after all threads are done.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
| |
Although unlikely the MHD_add_response_header can fail for
various reasons. If it fails something odd is going on.
So check we can actually add a response header and log an
error if we cannot.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
| |
The intern() function called set_metrics() outside a necessary lock
being held. helgrind identified this race condition. No QA impact.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
| |
gcc address sanitizer detected a leak of the debuginfod_client
winning_headers when the handle was reused. Make sure to free and
reset the winning_headers field before reuse.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
|
| |
In a couple of places we might leak some memory when we encounter
an error. tmp_url might leak if realloc failed. escaped_string might
leak when setting up the data handle fails and we don't use it.
And one of the goto out1 should have been goto out2 to make sure
we release all allocated resources on exit (also updated a wrong
comment about that).
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
| |
gcc address sanitizer detected a read after the end of string in
sqlite3_sharedprefix_fn. Make sure to stop comparing the strings when
seeing the zero terminator.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
| |
Since we are multi-threaded using gmtime might cause a data race
because gmtime reuses a global struct to write data into. Make
sure that each thread uses their own struct tm and use gmtime_r
instead.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
| |
"system.h" only declares the error() function, so it needs to be in an
'extern "C"' block, otherwise linking fails.
Since we are here, use quotes for "system.h" header, since it's a local
header, not a system one.
Signed-off-by: Érico Nogueira <erico.erc@gmail.com>
|
| |
|
|
|
|
|
|
| |
Add support for a limited mode for debuginfod that uses a pure
read-only sqlite index. This mode is useful for load spreading based
on naively shared or replicated databases.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For large databases and many stale files, it was possible to starve
rescan operations by numerous groom "nuke" (database delete ops).
Under the theory that including new data is at least as important as
aging old, we now impose a rough deadline on groom queries.
In the process, we discovered that we were commiting some
undefined-behaviour sqlite ops (deleting rows while iterating), which
may explain some previous heisenbug occurrences. So the groom nuke
operations are split into decision & action phases, with associated
progress-tracking metrics.
Testing the timeout facility requires hand-testing beyond the
testsuite (since it requires LARGE databases to show measurable query
times). So confirmed this part by hand.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Negative cache (000-permission) files were incorrectly treated as
valid cached files for the root user, because root can open even
000-perm files without -EACCES. Corrected this checking sequence.
Fixed the debuginfod testsuite to run to completion as root or
as an ordinary user, correcting corresponding permission checks:
stat -c %A $FILE
is right and
[ -w $FILE] [ -r $FILE ]
were wrong.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
Make sure we don't use any of the more experimental protocols
libcurl might support. URLs can be redirected and we might want
to follow http -> https, but not e.g. gopher or pop3.
Suggested-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This patch aims to extend http_responses_* metrics with another label
"type" by getting the extra artifact-type content added as a new
key=value tag.
v2, tweaked patch to perform artifact-type sanitization at point of
vulnerability rather than in general metric tabulation logic.
Signed-off-by: Di Chen <dichen@redhat.com>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
While checking PR27277 on some buildbots, greping would fail in
run-debuginfod-response-headers.sh. This was because querying the
debuginfod server occurs after checking if the responseh headers had
arrived, leaving the possibility to leave the querying loop before
outputting the headers which caused the grep failure. Querying now
occurs before checking if response headers have arrived, so that they
will certainly be printed and grep will find them.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allow users, with enough verbosity, to print the HTTP response headers
upon retrieving a file. These files may include several custome http
response headers such as X-DEBUGINFOD-FILE, X-DEBUGINFOD-SIZE, and
X-DEBUGINFOD-ARCHIVE. These headers are added from the daemon, in
debuginfod.cxx.
E.g output:
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 4095072
Cache-Control: public
Last-Modified: Thu, 09 Sep 2021 19:06:40 GMT
X-FILE: debuginfod
X-FILE-SIZE: 4095072
Content-Type: application/octet-stream
Date: Fri, 10 Sep 2021 16:38:06 GMT
https://sourceware.org/bugzilla/show_bug.cgi?id=27277
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, urls containing '/', so most urls, would escape '/' to %2F,
which is undesirable for use in other libraries which may escape
differently. This patch escapes the '/' and replaces all of them
ensuring there are no %2Fs sent.
Some inefficiencies within the code were fixed, such as changing constant
operations of a while loop within a for loop to a while loop outside of
a for loop. Also strlen is no longer used within the loop, simplifying
the interior operations to mere arithmetic.
https://sourceware.org/bugzilla/show_bug.cgi?id=28034
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
debuginfod's scan and groom operations (thread_main_scanner,
thread_main_fts_source_paths) are intended to be mutually exclusive,
as a substitute for more complicated sql transaction batching. (This
is because scanning / grooming involves inserting or deleting data
from multiple related tables.)
The workq class that governs this in debuginfod.cxx has a problem: if
the workq just becomes empty, its sole entry pulled by a scanner
thread in response to a wait_front(), an 'idler' groomer thread is
ALSO permitted to run, because there is no indication as to when the
scanner thread operation finishes, only when it starts.
Extending the workq with a counter ("fronters") to track any active
scanning activity (even if the workq is empty) lets us block idlers
groomers a little longer.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
| |
Without the endl the next log message will not start on its own line.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
| |
Return values of functions returning "void *", e.g. calloc, malloc,
realloc, xcalloc, xmalloc, and xrealloc, do not need explicit casts.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
debuginfod opens the database twice, once in read/wrote and once in
read-only mode. This means the magic ":memory:" in-memory database
cannot be used as is because the two connections don't really share
the underlying database. Fix this by turning ":memory:" into
":file::memory:?cache=shared" which makes the in-memory database
shared. See https://sqlite.org/inmemorydb.html
Document this in debuginfod.8 and make some tests use -d :memory:
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If someone misconfigures a debuginfod federation to have loops, and
a nonexistent buildid lookup is attempted, bad things will happen,
as is documented.
This patch aims to reduce the risk by adding an option to debuginfod
that functions kind of like an IP packet's TTL: a limit on the length of
XFF: header that debuginfod is willing to process. If X-Forwarded-For:
exceeds N hops, it will not delegate a local lookup miss to upstream
debuginfods.
Commit ab38d167c40c99 causes federation loops for non-existent resources
to result in multiple temporary deadlocks, each lasting for
$DEBUGINFOD_TIMEOUT seconds. Since concurrent requests for each unique
resource are now serialized, federation loops can result in one server
thread waiting to acquire a lock while the server thread holding the
lock waits for the first thread to respond to an http request.
This PR can help protect against the above multiple temporary deadlocks
behaviour. Ex. if --forwarded-ttl-limit=0 then the timeout behaviour of
local loops should be avoided.
https://sourceware.org/bugzilla/show_bug.cgi?id=27917
Signed-off-by: Di Chen <dichen@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
`error.h`'s inclusion was centralised into the `system.h` header. As
the implementation currently includes `system.h` already, the inclusion
of `error.h` is unnecessary. This prepares for a future portability
change to allow elfutil to build with alternate libc implementations.
Signed-off-by: Saleem Abdulrasool <abdulras@google.com>
|
| |
|
|
|
|
|
|
|
|
| |
The initial code for bug #27673 accidentally nuked all buildid service
concurrency, not just identical concurrent requests. Correct this
with one-liner patch. Observing the effect in the automated testsuite
is difficult, so hand-tested against large requests and short ones,
run in an interleaved way.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
DEBUGINFOD_TIMEOUT is a good way to catch servers that are too slow to
*start* transmitting a file. But we have no way of limiting total
download time or space. A user might prefer to have his debugger fetch
only quick & small files, and make do without the bigger ones. Some
transitive dependencies of e.g. gnome programs are huge: 3GB of LLVM
debuginfo, 1GB of webkitgtk, etc. etc.
DEBUGINFOD_MAXSIZE and DEBUGINFOD_MAXTIME were added to dictate the
max download size and time of a debuginfod client. DEBUGINFOD_MAXSIZE
is handled server-side and is sent using the http header:
X-DEBUGINFOD-MAXSIZE. The client side then checks to ensure this maxsize
has been respected.
https://sourceware.org/bugzilla/show_bug.cgi?id=27982
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Gazing at server logs, one sees a minority of clients who appear to have
duplicate query traffic coming in: the same URL, milliseconds apart.
Chances are the user accidentally doubled her $DEBUGINFOD_URLS somehow,
and the client library is dutifully asking the servers TWICE. Bug #27863
reduces the pain on the servers' CPU, but dupe network traffic is still
being paid. We should reject sending outright duplicate concurrent
traffic.
The urls are now simply removed upon finding a duplicate after url
construction.
https://sourceware.org/bugzilla/show_bug.cgi?id=27983
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When requesting some source files, some URL-inconvenient chars
sometimes pop up. Example from f33 libstdc++:
/buildid/44d8485cb75512c2ca5c8f70afbd475cae30af4f/source/usr/src/debug/
gcc-10.3.1-1.fc33.x86_64/obj-x86_64-redhat-linux/x86_64-redhat-linux/
libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/
condition_variable.cc
As this URL is passed into debuginfod's handler_cb, it appears that the
+ signs are helpfully unescaped to spaces by libmicrohttpd, which
'course breaks everything.
In order to ensure the server properly parses urls such as this one,
%-escape characters on the client side so that the correct url
is preserved and properly processed on the server side.
https://sourceware.org/bugzilla/show_bug.cgi?id=28034
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The debuginfod fdcache-prefetch logic has been observed to show some
degeneracies in operation. Since fdcache evictions are done
frequently, and freshly prefetched archive elements are put at the
back of lru[], each eviction round can summarily nuke things that
were just prefetched .... and are just going to be prefetched again.
It would be better to have two lru lists, or being able to insert
newly prefetched entries somewhere in the middle of the list rather
than at the very very end.
https://sourceware.org/bugzilla/show_bug.cgi?id=25978
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
In debuginfod-client.c (debuginfod_query_server),insert a
goto statement for jumping back to the beginning of curl
handles set up if query fails and a non ENOENT error is returned.
Also introduced DEBUGINFOD_RETRY_LIMIT_ENV_VAR and default
DEBUGINFOD_RETRY_LIMIT(which is 2).
Correponding test has been added to tests/run-debuginfod-find.sh
Signed-off-by: Alice Zhang <alizhang@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The debuginfod -I/-X regexes operate during traversal to identify
those files in need of scanning. The regexes are not used during
grooming. This means that if from run to run, the regex changes so
that formerly indexed files are excluded from traversal, the data is
still retained in the index.
This is both good and bad. On one hand, if the underlying data is
still available, grooming will preserve the data, and let clients ask
for it. On the other hand, if the growing index size is a problem,
and one wishes to age no-longer-regex-matching index data out, there
is no way.
Let's add a debuginfod flag to use regexes during grooming.
Specifically, in groom(), where the stat() test exists, also check
for regex matching as in scan_source_paths(). Treat failure of the
regex the same way as though the file didn't exist.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
|
| |
|
|
|
|
|
| |
When the calloc call in debuginfod_begin fails we should skip all
initialization of the client handle.
Signed-off-by: Mark Wielaard <mark@klomp.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sometimes, due to configuration error, mishap, or DoS misadventure, a
debuginfod server may receive near-concurrent requests for the exact
same data from multiple clients. In practically all cases, it is
beneficial to the clients, as well as the server, to serialize these
requests. This way, debuginfod does not waste CPU in repeatedly &
concurrently decompressing large archives or querying upstream
servers. Second and later requesters can benefit from the fdcache /
client-cache and get their results, probably earlier!
This patch adds an "after-you" queueing phase to servicing
http-buildid requests, whereby thereads serialize themselves on each
query URL being serviced at the moment. Prometheus metrics are added,
and the http GET trace line is modified to print the queue+service
times separately.
Hand-tested on large kernel-debuginfo's, and shows host CPU refusing
to multiply in the face of concurrent identical queries. The
automated test tries a hundred concurrent curls, at least some of
which are slow enough to trigger the "after-you" wait here.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
PR27701 implemented curl handle reuse in debuginfod_client objects,
but with an unexpected bug. Server responses returning an error
"latched" because the curl_easy handles for error cases weren't all
systematically removed from the curl multi handle. This prevented
their proper re-addition the next time.
This version of the code simplfies matters by making only the curl
curl_multi handle long-lived. This turns out to be enough, because it
can maintain a pool of long-lived http/https connections and related
data, and lend them out to short-lived curl_easy handles. This mode
handles errors or hung downloads even better, because the easy handles
don't undergo complex state transitions between reuse.
A new test case confirms this correction via the federating debuginfod
instance (cleaning caches between subtests to make sure http* is being
used and reused).
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
| |
|
|
|
|
|
| |
New function in system.h that returns true if a string has a given
prefix, false otherwise. Use it in place of strncmp.
Signed-off-by: Martin Liška <mliska@suse.cz>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add debuginfod_config_cache for reading and writing to cache
configuration files, make use of the function within
debuginfod_clean_cache and debuginfod_query_server.
In debuginfod_query_server, create 000-permission file on failed
queries. Before querying each BUILDID, if corresponding 000 file
detected, compare its stat mtime with parameter from
.cache/cache_miss_s. If mtime is fresher, then return ENOENT and
exit; otherwise unlink the 000 file and proceed to a new query.
tests: add test in run-debuginfod-find.sh
test if the 000 file is created on failed query; if querying the
same failed BUILDID, whether the query should proceed without
going through server; set the cache_miss_s to 0 and query the same
buildid, and this time should go through the server.
Signed-off-by: Alice Zhang <alizhang@redhat.com>
|
