summaryrefslogtreecommitdiff
path: root/debuginfod
Commit message (Collapse)AuthorAgeFilesLines
* debuginfod: Accelerate traversal, shutdowns, improve metricsFrank Ch. Eigler2020-10-312-33/+57
| | | | | | | | | | | | | Added new metrics for scanning that allow estimation of its reading bandwidth. Accelerated responsivity to SIGINT shutdown during archive-scanning phase, which previously insisted on completely processing the current archive. Noted in systemd service file that in the worst case, it might still take a long time. Accelerated traversals by moving regex -I/-X handling to apply to file names only (as always documented), so directory traversal metrics are accurate regardless of their name. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR26775: restore thread_work_groom metric to cycle countFrank Ch. Eigler2020-10-302-14/+40
| | | | | | | | | ... and add new metrics about progress of traversal and groom processes. Correct one control flow abnormality that could prematurely end a scanner thread and might have accounted for the inconsistent test results from the previous patch. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR26775: make grooming progress visible & interruptibleFrank Ch. Eigler2020-10-302-3/+36
| | | | | | | | | | | On very large servers, it's desirable to be able to interrupt a rescan or groom cycle. SIGUSR[12] now do that. (Unfortunately, this is not practically testable in the testsuite, since these cycles are so fast on that small dataset.) We also expose more internal progress count about the grooming pass, so the administrator can assess possible need to interrupt. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR26810: debuginfod should tolerate some absence/renaming sans groomingFrank Ch. Eigler2020-10-302-15/+25
| | | | | | | | | | debuginfod now knows to handle a case where a buildid search is satisfiable from more than one source (e.g., archive location), but some of them are invalid. New exception catching beneath the sqlite scanning loop ensures all possible matches are scanned in case of errors. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod: Translate CURLE_PEER_FAILED_VERIFICATION to ECONNREFUSED.Mark Wielaard2020-10-262-0/+6
| | | | | | | | | | | When a file couldn't be retrieved because of an bad HTTPS certificate find-debuginfod currently says: Server query failed: No such file or directory With this patch it will say: Server query failed: Connection refused Signed-off-by: Mark Wielaard <mark@klomp.org>
* PR26756: add more prometheus metrics to debuginfodFrank Ch. Eigler2020-10-212-16/+61
| | | | | | | | | Add an error_count{} family of metrics for each libc/libarchive/http exception instance created during operation. Add a family of fdcache* metrics for tracking fdcache operations and status. Test via a injecting a permission-000 empty nothing.rpm in the testsuite. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod: suppress fdcache prefetching during dwz lookupFrank Ch. Eigler2020-10-212-7/+18
| | | | | | | | | | | | | During a recent from-scratch reindexing of the rpm/deb corpus at debuginfod.elfutils.org, we found the fdcache chewed up an abnormal amount of $TMPDIR space. This was due to internal .dwz lookups, which triggered fdcache prefetching as for a webapi query, but there was not a timely fdcache eviction pass to clean it up again. Rather than add that pass, it's better to suppress the prefetching completely, as an internal .dwz search will only ever need that file, not any others from the same archive. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod: store only canonicalized sref pathnames in databaseFrank Ch. Eigler2020-09-182-46/+28
| | | | | | | | | | | Since PR25548, we let debuginfod answer /buildid/HEX/source/PATH queries with both canonicalized and raw PATHs. It canonicalizes incoming paths, but still stored the raw paths in the database too. This near-dupe storage is not needed, since the queries would always find the canonicalized version too, so stop doing that. This saves database space/time. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod: Fix BUILD_STATIC build.Mark Wielaard2020-09-172-2/+7
| | | | | | | | The BUILD_STATIC build, as used by the coverage setup, was broken for debuginfod because the DUMMY_LIBDEBUGINFOD case was mixed up. It should include libcurl_LIBS when NOT doing a dummy build. Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod-find.c: Fix license block comment. File is GPLv3+ only.Mark Wielaard2020-09-162-3/+7
| | | | | | | The license itself was correct, just the comment to get a copy of the LGPL was wrong. Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod-find: Support compressed (kernel) ELF images.Mark Wielaard2020-09-152-2/+7
| | | | | | | | | By using dwelf_elf_begin instead of elf_begin we automatically get support for determining the build-id of compressed (kernel) images. https://sourceware.org/bugzilla/show_bug.cgi?id=26599 Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod: DEBUGINFOD_URLS should accept scheme-free urlsAlice Zhang2020-07-052-2/+32
| | | | | | | | | | | | Check scheme instead of effective url so that user may abbreviate DEBUGINFOD_URL. Add one test for scheme free http url. Notice that libcurl does not provide an almighty scheme free url support, /path/to/something without FILE:// can not be recognized in most circumstances, therefore for the neatness of our code structure, DEBUGINFOD_ URL of scheme "FILE" must be input as URI. Signed-off-by: Alice Zhang <alizhang@redhat.com>
* debuginfod: Add --disable-libdebuginfod and --enable-libdebuginfod=dummy.Mark Wielaard2020-07-043-4/+73
| | | | | | | | | | Make it possible to build just the debuginfod client or to create a dummy libdebuginfod that doesn't link against libcurl. The dummy library can be used for bootstrapping. For testing purposes you can also build debuginfod against the dummy libdebuginfod but then the debuginfod server will not be able to do delegation. Signed-off-by: Mark Wielaard <mark@klomp.org>
* PR26195: adapt debuginfod to API change in libmicrohttpd-0.9.71Frank Ch. Eigler2020-07-021-3/+15
| | | | | | | | To make our code build with -Werror as well as against older libmicrohttpd, we must conditionalize the data type (int vs. enum) returned by callbacks and some mhd functions. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod: Make sure handle_data can be allocated and is always freed.Mark Wielaard2020-06-242-8/+19
| | | | | | | | | When allocating handle_data we should check for out of memory failures. Also when the allocation has succeeded make sure we always clean up by going to out1 on any future errors. So move the curl_multi_init call earlier, because that goes to out0 on failure. Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod: Fix build_id hexadecimal length check.Mark Wielaard2020-06-242-1/+6
| | | | | | | | | | When is debuginfod_query_server is given an hexadecimal string as build-id build_id_len will be zero. We were checking the size of the build_id_bytes destination string instead of the string length of build_id input string. Make sure the input string is not too big or strcpy might overwrite then end of the build_id_bytes array. Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod: Make sure suffix can place zero terminator when copying filenameMark Wielaard2020-06-242-2/+7
| | | | | | | | | We need to make sure that we can always place a zero terminator at the end of suffix when we are copying the filename. So add one more char to the suffix array. And make sure that we can always add an extra escape character when we need to escape the current character. Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod: Handle not being able to fopen interval_path.Mark Wielaard2020-06-242-2/+12
| | | | | | | | Although we check for and/or create the interval_path right before, there is still a possibility that the fopen call fails. Handle that as if the file is unreadable. Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod: Document and sanity check debuginfod_add_http_header format.Mark Wielaard2020-03-302-0/+15
| | | | | | | | | | Document and sanity check the format of the header string form that can be passed to debuginfod_add_http_header. It should contain precisely one colon, which cannot be the first or last character. And the function should only be used to add optional headers, not replace any existing standard ones. Anything else isn't supported. Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod-client default_progressfn: formatting fixFrank Ch. Eigler2020-03-292-1/+17
| | | | | | | | | | The saga of clean $DEBUGINFOD_PROGRESS=1 output continues. Previous code would sometimes insert a \n (a blank line) into the output stream, even if the target file was found in a cache and thus the progressfn was never called. New code sets a client flag to track this case more precisely. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25728: disable debuginfod --port=0Mark Wielaard2020-03-282-1/+6
| | | | | | | | | | | | When starting debuginfod with --port=0 it would start using a random port (possibly different for ipv4 and ipv6). This seems to be an accidental and not very useful functionality. Just produce an error when started with --port=0. https://sourceware.org/bugzilla/show_bug.cgi?id=25728 Signed-off-by: Mark Wielaard <mark@klomp.org> Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25739: debuginfod correct mtime for fdcache'd filesFrank Ch. Eigler2020-03-282-0/+13
| | | | | | | | | Files extracted from archives then left in the fdcache need to get get their mtime set consistently, so that a subsequent cache-hit fetch can relay the correct mtime to clients. Reported-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25722: debuginfod-find: accept /path/names in place of buildid hexFrank Ch. Eigler2020-03-273-7/+70
| | | | | | | | | Extend the debuginfod-find command line interface to permit /file/names instead of only buildid hexadecimal strings. v2: move code into debuginfod-find; client API remains buildid only. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod: User-Agent and X-Forwarded-For header relayFrank Ch. Eigler2020-03-275-12/+87
| | | | | | | | | Extend the debuginfod client API with a function to stuff outgoing headers into libcurl http transfers. Use this from debuginfod so federated trees of debuginfod/httpd can trace back to to the originating client for administrative purposes. Docs & test included. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25448: debuginfod: add transfer performance metricsFrank Ch. Eigler2020-03-272-0/+17
| | | | | | | | | | | | | | | | | | | | | We now export metrics related to the time taken and data sent, from which prometheus type tools can compute nice time series with averages. http_responses_duration_milliseconds_count{code="200"} 63 http_responses_duration_milliseconds_count{code="404"} 2 http_responses_duration_milliseconds_count{code="503"} 1 http_responses_duration_milliseconds_sum{code="200"} 66 http_responses_duration_milliseconds_sum{code="404"} 2 http_responses_duration_milliseconds_sum{code="503"} 0 http_responses_transfer_bytes_count{code="200"} 63 http_responses_transfer_bytes_count{code="404"} 2 http_responses_transfer_bytes_count{code="503"} 1 http_responses_transfer_bytes_sum{code="200"} 425177 http_responses_transfer_bytes_sum{code="404"} 18 http_responses_transfer_bytes_sum{code="503"} 37 Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25583: debuginfod: conditionally prefer bsdtar to dpkg for .deb handlingFrank Ch. Eigler2020-03-262-2/+15
| | | | | | | Prefer /usr/bin/dpkg-deb if installed, as normal on a debian system. Suggested-by: Mark Wielaard <mark@klomp.org> Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25583: debuginfod: prefer bsdtar to dpkg for .deb handlingFrank Ch. Eigler2020-03-262-2/+8
| | | | | | | | It turns out a bsdtar subshell can do the job of dpkg-deb. bsdtar comes from/with libarchive so it should be available everywhere. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25548: CURLOPT_PATH_AS_IS backward compatibilityFrank Ch. Eigler2020-03-261-1/+1
| | | | | | | old libcurl lacks the CURL_AT_LEAST_VERSION macro, so use LIBCURL_VERSION_NUM testing instead. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25548: CURLOPT_PATH_AS_IS backward compatibilityFrank Ch. Eigler2020-03-262-0/+10
| | | | | | | | libcurl < 7.42 lacks the CURLOPT_PATH_AS_IS flag, but extraneous client-side canonicalization is mostly harmless. Signed-off-by: Frank Ch. Eigler <fche@redhat.com> Reported-by: Mark Wielaard <mark@klomp.org>
* PR25548: support canonicalized source-path names in debuginfod webapiFrank Ch. Eigler2020-03-263-1/+135
| | | | | | | | | | | | | Programs are sometimes compiled with source path names containing noise like /./ or // or /foo/../, and these survive into DWARF. This code allows either raw or canonicalized pathnames in the webapi, by letting the client pass things verbatim, and letting the server store/accept both raw and canonicalized path names for source files. Tests included & docs updated. Signed-off-by: Frank Ch. Eigler <fche@redhat.com> Reported-by: Eli Schwartz <eschwartz@archlinux.org> Tested-by: Eli Schwartz <eschwartz@archlinux.org>
* PR25367: improve debuginfod webapi loggingFrank Ch. Eigler2020-03-262-10/+53
| | | | | | | Improve debuginfod logging to show webapi query results including http status, sizes, and processing times. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod-client thinko: non-default progressfn extra outputFrank Ch. Eigler2020-03-242-34/+42
| | | | | | | | A previous commit changed the default_progressfn output format to \rFOOBAR, to be terminated by an \n when the download finished. The \n terminator was conditional on the wrong thing (env var setting, rather than actual progressfn setting), so the \n could be printed even if an app overrode the default.
* debuginfod-find: Correct error check for -v source comboFrank Ch. Eigler2020-03-242-2/+7
| | | | | | | Hardcoding argv[2] is wrong in presence of -v option. Code immediately following did the correct argv[] indexing already. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod client API: add get_url functionFrank Ch. Eigler2020-03-225-10/+81
| | | | | | | This function lets a client know, during or after a progressfn callback, what the url of the winning outgoing download is/was. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod client API: add get/set user_data functionsFrank Ch. Eigler2020-03-225-4/+40
| | | | | | | | Add a pair of functions to associate a void* parameter with a client object. Requested by GDB team as a way to pass file names and such user-interface data through to a progressfn callback. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod-client: Update cache_path when the new default path existsAaron Merey2020-03-042-9/+12
| | | | | | | | | Update cache_path with the path of the new default directory when this directory already exists. Previously cache_path was updated only when creating the new default directory and would otherwise be set to the old default path. Signed-off-by: Aaron Merey <amerey@redhat.com>
* debuginfod-client: default to XDG cache.Aaron Merey2020-02-282-26/+100
| | | | | | | | | | | | PR25502: debuginfod client should default to XDG cache Location of client cache now defaults to $XDG_CACHE_HOME/debuginfod_client. If XDG_CACHE_HOME is not set then fallback to $HOME/.cache/debuginfod_client. Also maintain backwards compatibility with the previous default path- if $HOME/.debuginfod_client_cache exists, use that instead of the new defaults. Signed-off-by: Aaron Merey <amerey@redhat.com>
* debuginfod: file:// URLs: handle curl resp. codeKonrad Kleine2020-02-262-12/+31
| | | | | | | | | | | | | | When file:// is used for DEBUGINFOD_URLS, then the response code for a successful server query is 0 and not 200. Using file:// can be helpful when you want to test your debuginfod-client integration against a mocked file tree that mimics the HTTP URLs from the debuginfod server. This way you don't have to run the debuginfod server at all. Fixes https://sourceware.org/bugzilla/show_bug.cgi?id=25600 Signed-off-by: Konrad Kleine <kkleine@redhat.com>
* debuginfod PR25583: map -R to -Z.rpmFrank Ch. Eigler2020-02-252-1/+5
| | | | | | | | | | It was reported that libarchive (bsdtar) at least as far back as rhel7 (3.1.2) can natively process RPM files, so there's no need to mediate those accesses through rpm2cpio. There's no noteworthy performance or testing impact. Signed-off-by: Frank Ch. Eigler <fche@redhat.com> Signed-off-by: Mark Wielaard <mark@klomp.org>
* PR25375: fdcache prefetching to reduce repeated archive decompressionFrank Ch. Eigler2020-02-252-23/+99
| | | | | | | Introduce new option --fdcache-prefetch to accelerate repeated queries from the same debuginfo archive. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25365: debuginfod-client: restrict cleanup to client-pattern filesAaron Merey2020-02-192-1/+17
| | | | | | | | Avoid deleting general files and directories in case a user mis-sets $DEBUGINFOD_CACHE_PATH or accidentally moves a file into the cache. Signed-off-by: Aaron Merey <amerey@redhat.com>
* debuginfod: archive processing: handle -Z EXT=cat with direct fopenMark Wielaard2020-02-101-10/+37
| | | | Signed-off-by: Mark Wielaard <mark@klomp.org>
* debuginfod: generalized archive supportFrank Ch. Eigler2020-02-102-10/+45
| | | | | | | | | Add a '-Z EXT[=CMD]' option to debuginfod, which lets it scan any given extension and run CMD on it to unwrap distro archives. For example, for arch-linux pacman files, -Z '.tar.zst=zstdcat' lets debuginfod grok debug and source content in split-debuginfo files. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* debuginfod.h: fix typo in comment: legnth -> lengthKonrad Kleine2020-02-101-1/+1
| | | | Signed-off-by: Konrad Kleine <kkleine@redhat.com>
* debuginfod: diagnostics verbosity tweakFrank Ch. Eigler2020-01-222-1/+7
| | | | | | | marxin reported "skipping hat= .... empty comp_dir" messages were too chatty for the journnal. Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
* PR25394 cont'd: debuginfod timing fix for fts-traversal threadFrank Ch. Eigler2020-01-201-0/+1
| | | | | The new code neglected to set the last_rescan timestamp, leading to overly frequent rescanning.
* PR25394 followup#2: debuginfod casting fixesFrank Ch. Eigler2020-01-201-2/+2
| | | | | Buildbot still reports type warnings in time_t arithmetic. Explicit (long)er cast pushed as obvious ... or is it? :-)
* PR25394 followup: debuginfod casting fixesFrank Ch. Eigler2020-01-201-2/+2
| | | | | Buildbot reports type warnings in time_t arithmetic. Explicit (long) cast pushed as obvious.
* PR25394: debuginfod mutex between grooming and scanningfche/debuginfod-PR25394Frank Ch. Eigler2020-01-192-44/+95
| | | | | | | | | | | | | | | | | | | | Extended the work-queue concept with "idlers" - other threads that block on the work queue until it becomes empty (rather than normal consumers that block on it until it becomes non-empty). Use this facility for the groomer thread to avoid working at the same time as the scanner threads. Use this for the fts traversal thread for similar reasons. One user-visible effect: response to SIGUSR1 and SIGUSR2 will wait until the work queue runs empty, but the man page was unspecific so does not need changing. It's not obvious how to test this with a tests/ dataset so small that scanning takes negligible time, so the former races are very tight. P.S. We also evaluated using sqlite level transactions to isolate the scanner thread groups-of-operations from the groomer. These experiments failed to produce a nominally concurrent debuginfod, having triggered "database locked" type errors. So we remain single-threaded (fully serialized) at the sqlite API level.
* PR25415: fix debuginfod SEGV for empty search path listFrank Ch. Eigler2020-01-182-0/+9
| | | | | Prevent triggering undefined behaviour of fts(3), which causes a SEGV on fedora glibc 2.29.
View Lorry Controller Status