PR29472: debuginfod: add metadata query webapi, C api, clientusers/fche/try-pr29472e

This patch extends the debuginfod API with a "metadata query"
operation.  It allows clients to request an enumeration of file names
known to debuginfod servers, returning a JSON response including the
matching buildids.  This lets clients later download debuginfo for a
range of versions of the same named binaries, in case they need to to
prospective work (like systemtap-based live-patching).  It also lets
server operators implement prefetch triggering operations for popular
but slow debuginfo slivers like kernel vdso.debug files on fedora.

Implementation requires a modern enough json-c library.

% debuginfod-find metadata file /bin/ls
% debuginfod-find metadata glob "/usr/local/bin/c*"

Documentation and testing are included.

Signed-off-by: Ryan Goldberg <rgoldber@redhat.com>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>

4 files changed, 65 insertions, 0 deletions

diff --git a/doc/ChangeLog b/doc/ChangeLog
index 7f2d6ff4..4ed30eb2 100644
--- a/doc/ChangeLog
+++ b/doc/ChangeLog
@@ -1,3 +1,10 @@
+2023-04-12  Ryan Golberg <rgoldberg@redhat.com>, Frank Ch. Eigler <fche@redhat.com>
+
+	PR29472: debuginfod metadata query
+	* debuginfod-find.1: Document metadata query.
+	* debuginfod-client-config.7: Document metadata cache control setting.
+	* debuginfod.8: Document new option and webapi.
+
 2023-02-14  Mark Wielaard  <mark@klomp.org>
 
 	* debuginfod.8: Add .TP before -g.
diff --git a/doc/debuginfod-client-config.7 b/doc/debuginfod-client-config.7
index 53d82806..bc98d1e6 100644
--- a/doc/debuginfod-client-config.7
+++ b/doc/debuginfod-client-config.7
@@ -134,3 +134,11 @@ are short-circuited (returning an immediate failure instead of sending
 a new query to servers).  This accelerates queries that probably would
 still fail.  The default is 600, 10 minutes.  0 means "forget
 immediately".
+
+.TP
+.B metadata_retention_s
+This control file sets how long to remember the results of a metadata
+query.  New queries for the same artifacts within this time window are
+short-circuited (repeating the same results).  This accelerates
+queries that probably would probably have the same results.  The
+default is 86400, 1 day.  0 means "do not retain".
diff --git a/doc/debuginfod-find.1 b/doc/debuginfod-find.1
index 7d577bab..6652539c 100644
--- a/doc/debuginfod-find.1
+++ b/doc/debuginfod-find.1
@@ -29,6 +29,8 @@ debuginfod-find \- request debuginfo-related data
 .B debuginfod-find [\fIOPTION\fP]... source \fIBUILDID\fP \fI/FILENAME\fP
 .br
 .B debuginfod-find [\fIOPTION\fP]... source \fIPATH\fP \fI/FILENAME\fP
+.br
+.B debuginfod-find [\fIOPTION\fP]... metadata \fIKEY\fP \fIVALUE\fP
 
 .SH DESCRIPTION
 \fBdebuginfod-find\fP queries one or more \fBdebuginfod\fP servers for
@@ -119,6 +121,36 @@ l l.
 \../bar/foo.c AT_comp_dir=/zoo/	source BUILDID /zoo//../bar/foo.c
 .TE
 
+.SS metadata \fIKEY\fP \fIVALUE\fP
+
+All designated debuginfod servers are queried for metadata about files
+in their index.  Different search keys may be supported by different
+servers.
+
+.TS
+l l l .
+KEY	VALUE	DESCRIPTION
+
+\fBfile\fP	path	exact match \fIpath\fP, including in archives
+\fBglob\fP	pattern	glob match \fIpattern\fP, including in archives
+.TE
+
+The results of the search are output to \fBstdout\fP as a JSON array
+of objects, supplying metadata about each match.  This metadata report
+may be cached.  It may be incomplete and may contain duplicates.  For
+each match, the result is a JSON object with these fields.  Additional
+fields may be present.
+
+.TS
+l l l .
+NAME	TYPE	DESCRIPTION
+
+\fBbuildid\fP	string	hexadecimal buildid associated with the file
+\fBtype\fP	string	one of \fBdebuginfo\fP or \fBexecutable\fP
+\fBfile\fP	string	matched file name, outside or inside the archive
+\fBarchive\fP	string	archive containing matched file name, if any
+.TE
+
 .SH "OPTIONS"
 
 .TP
diff --git a/doc/debuginfod.8 b/doc/debuginfod.8
index 07cb01ae..1070d290 100644
--- a/doc/debuginfod.8
+++ b/doc/debuginfod.8
@@ -134,6 +134,14 @@ service load.  Archive pattern options must still be given, so
 debuginfod can recognize file name extensions for unpacking.
 
 .TP
+.B "\-\-metadata\-maxtime=SECONDS"
+Impose a limit on the runtime of metadata webapi queries.  These
+queries, especially broad "glob" wildcards, can take a large amount of
+time and produce large results.  Public-facing servers may need to
+throttle them.  The default limit is 5 seconds.  Set 0 to disable this
+limit.
+
+.TP
 .B "\-D SQL" "\-\-ddl=SQL"
 Execute given sqlite statement after the database is opened and
 initialized as extra DDL (SQL data definition language).  This may be
@@ -384,6 +392,16 @@ The exact set of metrics and their meanings may change in future
 versions.  Caution: configuration information (path names, versions)
 may be disclosed.
 
+.SS /metadata?key=\fIKEY\fP&value=\fIVALUE\fP
+
+This endpoint triggers a search of the files in the index plus any
+upstream federated servers, based on given key and value.  If
+successful, the result is a application/json textual array, listing
+metadata for the matched files.  See \fIdebuginfod-find(1)\fP for
+documentation of the common key/value search parameters, and the
+resulting data schema.
+
+
 .SH DATA MANAGEMENT
 
 debuginfod stores its index in an sqlite database in a densely packed