2021-02-04 Frank Ch. Eigler PR27092 low-memory handling * debuginfod.cxx (fdcache_mintmp): New parameter, with cmd-line option. (parse_opt): Parse it. (main): Default it. (statfs_free_enough_p): New function. (libarchive_fdcache::*): Call it to trigger emergency fdcache flush. (thread_main_scanner): Call it to report filesystem fullness metrics. (groom): Ditto. (set/add_metric): Take double rather than int64_t values. (archive_exception): Propagate suberror to metric label. (main): Detect pthread creation fatal errors properly. 2021-02-02 Frank Ch. Eigler PR27323 * debuginfod.cxx (dbq): New read-only database connection for queries only. (signal_handler): Interrupt it. (main): Open / close it. (handle_buildid): Use it for webapi queries only. (database_stats_report): Make more interruptible. Report sqlite3 operation times to the prometheus metrics. (groom): Make more interruptible. (thread_main_fts_source_paths, thread_main_groom): Ensure state/progress metrics are fresh even in case of exceptions. 2020-12-20 Dmitry V. Levin * .gitignore: New file. 2020-12-12 Dmitry V. Levin * debuginfod-client.c (debuginfod_query_server): Fix spelling typos in comments. * debuginfod.cxx: Likewise. (parse_opt): Fix spelling typos in error diagnostics. 2020-12-08 Dmitry V. Levin * Makefile.am [LIBDEBUGINFOD]: Create libdebuginfod.so.1 first, turn libdebuginfod.so into symlink. 2020-11-30 Dmitry V. Levin * Makefile.am (libdebuginfod.so): Replace $@.$(VERSION) with $(LIBDEBUGINFOD_SONAME). (install, uninstall, MOSTLYCLEANFILES): Replace libdebuginfod.so.$(VERSION) with $(LIBDEBUGINFOD_SONAME). (VERSION): Remove. * debuginfod.h: Rename to ... * debuginfod.h.in ... this. (DEBUGINFOD_SONAME): New macro. 2020-11-30 Dmitry V. Levin * Makefile.am (libdebuginfod.so$(EXEEXT)): Drop $(EXEEXT) suffix. 2020-11-25 Frank Ch. Eigler * debuginfod.cxx (step_ok_done): Correct typo in prom metric label. 2020-11-25 Frank Ch. Eigler * debuginfod.cxx (tmp_ms_metric): Switch from gettimeofday to clock_gettime(CLOCK_MONOTONIC) for time-interval measurements. (handler_cb, scan_source_paths, groom): Ditto. 2020-11-23 Frank Ch. Eigler * debuginfod.cxx (tmp_ms_metric): New class for RAII timing metrics. (sqlite_ps::reset, step*): Call it to track sqlite3 performance. (sqlite_exception ctor): Increment sqlite3 error_count. 2020-11-23 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Initialize struct handle_data errbuf to the empty string. 2020-11-11 Mark Wielaard * debuginfod-client.c (debuginfod_set_verbose_fd): New function. (struct debuginfod_client): Add verbose_fd. (struct handle_data): Add errbuf. (debuginfod_query_server): Produce verbose output when debuginfod_client verbose_fd is set. Only clear old data and set default_headers when any work is done. Always goto out when setting rc to an error value. Use CURLOPT_ERRORBUFFER to get more error output when verbose output is requested. * debuginfod.h (DEBUGINFOD_VERBOSE_ENV_VAR): New. (debuginfod_set_verbose_fd): Added. * debuginfod-find.c (parse_opt): Set debuginfod_set_verbose_fd on -v. * bdebuginfod.map (ELFUTILS_0.183): New section, add debuginfod_set_verbose_fd. 2020-11-21 Mark Wielaard * debuginfod.cxx (handle_root): New function. (handler_cb): Handle "/" and report url1 in webapi error. 2020-11-11 Mark Wielaard * debuginfod-find.c (progressfn): Use clock_gettime to print Progress at most 5 times a second. 2020-11-19 Frank Ch. Eigler * debuginfod.cxx (tmp_inc_metric): New class. (handler_cb): Use it to track webapi operations. 2020-11-01 Érico N. Rolim * debuginfod-client.c (debuginfod_init_cache): Use ACCESSPERMS for mkdir, DEFFILEMODE for open with O_CREAT. 2020-11-01 Érico N. Rolim * debuginfod.cxx: include libintl.h. 2020-11-01 Érico N. Rolim * Makefile.am (debuginfod_LDADD): Add argp_LDADD and fts_LIBS. (debuginfod_find_LDADD): Likewise. (libdebuginfod_so_LDLIBS): Add fts_LIBS. 2020-10-31 Frank Ch. Eigler * debuginfod.cxx (scan_source_file, scan_archive_file): Add new scanned_bytes_total, scanned_files_total metrics. (archive_classify): Exit early if interrupted. (scan_source_paths): Perform realpath/regex checks only on FTS_F files. Tweak metrics. 2020-10-30 Frank Ch. Eigler PR26775 cont'd. * debuginfod.cxx (thread_main_scanner): Ensure control doesn't leave infinite loop until program exit, even if SIGUSR2. (scan_source_paths): Have traverser clean scanq on SIGUSR2. Emit additional traversed_total metrics. (groom): Emit additional groomed_total metrics. (thread_main_groom): Restore previous thread_work_total metric. 2020-10-29 Frank Ch. Eigler PR26775 * debuginfod.cxx (forced_*_count): Make these global. (runq::clear): New function. (thread_main_scanner): Check for pending SIGUSR2; interrupt. (scan_source_paths): Check for pending SIGUSR2; interrupt. (groom): Report prometheus stats before groom also. Check for pending SIGUSR1; interrupt. Increment thread_work_total for each file scanned, not the entire cycle. 2020-10-29 Frank Ch. Eigler PR26810 * debuginfod.cxx (handle_buildid_*_match): Throw exceptions for more lower level libc errors. (handle_buildid_match): Catch & report exceptions but return 0 for continued iteration in the caller. 2020-10-25 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Translate CURLE_PEER_FAILED_VERIFICATION to ECONNREFUSED. 2020-10-20 Frank Ch. Eigler PR26756: more prometheus metrics * debuginfod.cxx (*_exception): Add counters for error occurrences. (fdcache::*): Add counters for fdcache operations and status. (fdcache::set_metric): New fn for overall stat counts. (fdcache::limit): ... allow metric-less use from dtors. 2020-10-20 Frank Ch. Eigler * debuginfod.cxx (handle_buildid*): Add a parameter for detecting internally-originated lookups for dwz resolution. 2020-09-18 Frank Ch. Eigler * debuginfod.cxx (scan_source_file, archive_classify): Store only canonicalized file names in sdef & sref records in the database. 2020-09-08 Mark Wielaard * Makefile.am (BUILD_STATIC): Include libcurl_LIBS in libdebuginfod when NOT DUMMY_LIBDEBUGINFOD. 2020-09-16 Mark Wielaard * debuginfod-find.c: Fix license block comment. 2020-09-15 Mark Wielaard * debuginfod-find.c (main): Use dwelf_elf_begin. 2020-07-03 Alice Zhang * debuginfod-client.c (debuginfod_query_server): Use strncasecmp to compare effective_url. Try CURLINFO_SCHEME as fallback. 2020-06-19 Mark Wielaard * Makefile.am (bin_PROGRAMS): Guard with DEBUGINFOD and LIBDEBUGINFOD. (debuginfod_LDADD): Remove libcurl. (libdebuginfod): When static and DUMMY_LIBDEBUGINFO remove libcurl. (noinst_LIBRARIES): Guard with LIBDEBUGINFOD. (AM_CPPFLAGS): Add -Wno-unused-parameter when DUMMY_LIBDEBUGINFOD. (pkginclude_headers): Guard with LIBDEBUGINFOD (libdebuginfod_so_LIBS): Likewise. (+libdebuginfod_so_LDLIBS): Likewise. (install): Likewise. (uninstall): Likewise. * debuginfod-client.c: Include dummy functions when DUMMY_LIBDEBUGINFOD. * debuginfod.cxx: Remove curl.h include. 2020-06-16 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Check malloc. Move curl_multi_init call before handle_data malloc call. 2020-06-16 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Replace sizeof build_id_bytes check with strlen build_id check. 2020-06-16 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Increase suffix array and prepare having to escape 1 character with 2. 2020-06-16 Mark Wielaard * debuginfod-client.c (debuginfod_clean_cache): Handle failing fopen (interval_path). 2020-03-29 Mark Wielaard * debuginfod-client.c (debuginfod_add_http_header): Check header contains precisely one colon that isn't the first or last char. 2020-03-29 Frank Ch. Eigler * debuginfod-client.c (struct debuginfod_client): Add a flag field for progressfn printing. (default_progressfn): Set it if printing \rsomething. (debuginfod_end): Terminate with \n if flag set, i.e., only if the default_progressfn was actually called. 2020-03-27 Mark Wielaard * debuginfod.cxx (parse_opt): Check port is not zero. 2020-03-28 Frank Ch. Eigler * debuginfod.cxx (handle_buildid_r_match): During archive extraction / fdcache prefetching, set the mtime of each file in the cache. 2020-03-27 Frank Ch. Eigler * debuginfod-find.c (main): Extract buildid from /binary/ if given instead of hex string. * Makefile.am: Add elfutils library prereqs for debuginfod-find. 2020-03-24 Frank Ch. Eigler * debuginfod.h, libdebuginfod.map: New functions for _add_url_header. * debuginfod-client.c (struct debuginfod_client): Add headers fields. (debuginfod_add_http_header): New client api to add outgoing headers. (add_default_headers): Renamed from add_extra_headers, skip if flag. (debuginfod_query_server): Pass accumulated headers to libcurl. (debuginfod_end): Clean accumulated headers. (debuginfod_find_*): Add default headers at this point. * debuginfod.cxx (handle_buildid): Add conn pointer. Use it to relay incoming UA and XFF headers to federated upstream debuginfods. 2020-03-26 Frank Ch. Eigler * debuginfod.cxx (handler_cb): Export two families of metrics for prometheus traffic analysis: response times and data amounts. 2020-03-26 Frank Ch. Eigler * debuginfod.cxx (parse_opt): For -U, prefer dpkg-deb after all if access(3)-able, fallback to bsdtar. 2020-03-25 Frank Ch. Eigler * debuginfod.cxx (parse_opt): Associate a bsdtar subshell with the .deb & .ddeb extensions, instead of dpkg-deb. 2020-03-26 Frank Ch. Eigler * debuginfod-client.c (debuginfod_query_server): Don't set CURLOPT_PATH_AS_IS on old curl. Mostly harmless. 2020-03-24 Frank Ch. Eigler * debuginfod-client.c (debuginfod_query_server): Set CURLOPT_PATH_AS_IS, to propagate file names verbatim. * debuginfod.cxx (canon_pathname): Implement RFC3986 style pathname canonicalization. (handle_buildid): Canonicalize incoming webapi source paths, accept either one. (scan_source_file, archive_classify): Store both original and canonicalized dwarf-source file names. 2020-03-24 Frank Ch. Eigler * debuginfod.cxx (handle_buildid): In case of federated fallback queries, handle errors analogously to local ENOENT/404. (handle_metrics): Return a size-of-response value. (handler_cb): Add code to time entire application-side processing stage + response sizes + http codes, so as to emit a complete httpd-flavoured log line for each webapi request. 2020-03-24 Frank Ch. Eigler * debuginfod-client.c (debuginfod_query_server): Print the default_progressfn terminating \n message only if that progressfn is actually set. 2020-03-24 Frank Ch. Eigler * debuginfod-find.c (main): Correct /source full-pathness check for "debuginfod-find -v source deadbeef /pathname" case. 2020-03-22 Frank Ch. Eigler * debuginfod-client.c (struct debuginfod_client): Add url field. (struct handle_data): Add client field as backpointer. (debuginfod_write_callback): Compute & save URL. (default_progressfn): Print front pieces of the URL. (debuginfod_query_server): Clear URL and cleanup after progressfn. * debuginfod-find.c (main): Print URL at transfer conclusion. 2020-03-22 Frank Ch. Eigler * debuginfod.h, libdebuginfod.map: New functions for _get/set_user(). * debuginfod-client.c: Implement them. * debuginfod-find.c: Include a token call just for testing them. 2020-03-03 Aaron Merey * debuginfod-client.c (debuginfod_query_server): Update cache_path even when new default path already exists. 2020-02-27 Aaron Merey * debuginfod-client.c (xalloc_str): New macro. Call asprintf with error checking. (debuginfod_query_server): Use XDG_CACHE_HOME as a default cache location if it is set. Replace snprintf with xalloc_str. 2020-02-26 Konrad Kleine * debuginfod-client.c (debuginfod_query_server): Handle curl's response code correctly when DEBUGINFOD_URLS begin with file:// 2020-02-25 Frank Ch. Eigler * debuginfod.cxx (parse_opt): Treat -R as if -Z.rpm . 2020-02-25 Frank Ch. Eigler * debuginfod.cxx (fdcache_prefetch): New parameter. (parse_opt): Parse it. (main): Default it. (fdcache::fd_size_mb): Change to double for accuracy. (fdcache::probe): New function. (fdcache::intern): New option to intern at end of LRU. (fdcache::lookup): Clean fdcache. (handle_buildid_r_match): Implement multi-stage archive parsing, with optional prefetching of extracted contents into the fdcache. 2020-02-19 Aaron Merey * debuginfod-client.c (debuginfod_clean_cache): Restrict cleanup to client-pattern files. 2020-02-05 Frank Ch. Eigler * debuginfod.cxx (argp options): Add -Z option. (canonicalized_archive_entry_pathname): New function for distro-agnostic file name matching/storage. 2020-01-22 Frank Ch. Eigler * debuginfod.cxx (dwarf_extract_source_paths): Don't print "skipping hat" messages at verbosity <=3, too noisy. 2020-01-19 Frank Ch. Eigler * debuginfod.cxx (scanq): Rework to let groomer/fts threads synchronize with an empty workqueue, and lock out workqueue consumers. (thread_groom): Adopt new scanq idle APIs to lock out scanners. (thread_main_fts_source_paths): Adopt new scanq idler API to avoid being restarted while scanners haven't even finished yet. (thread_main_*): Increment thread_work_total metric only after a work cycle is completed, not when it begins. 2020-01-18 Frank Ch. Eigler * debuginfod.cxx (thread_main_scanner): Handle empty source_paths[]. 2020-01-11 Frank Ch. Eigler * debuginfod.cxx (libarchive_fdcache): New class/facility to own a cache of temporary files that were previously extracted from an archive. If only it could store just unlinked fd's instead of filenames. (handle_buildid_r_match): Use it to answer dwz/altdebug and webapi requests. (groom): Clean it. (main): Initialize the cache control parameters from heuristics. Use a consistent tmpdir for these and tmp files elsewhere. 2020-01-11 Frank Ch. Eigler * debuginfod.cxx (conninfo): Print User-Agent and X-Forwarded-For request headers, after mild safety-censorship (for easier machine processing). 2020-01-11 Frank Ch. Eigler * debuginfod.cxx: Rework threading model. (workq): New class for concurrent work-queue. (semaphore): Removed class, now unused. (scan_source_file_path): Rework into ... (scan_source_file): New function. (thread_main_scan_source_file_path): Nuke. (scan_source_archive_path): Rework into ... (scan_archive_file): New function. (thread_main_scanner): New function for scanner threads. (thread_main_fts_source_paths): New function for traversal thread. (scan_source_paths): ... doing this. (thread_groom): Tweak metrics for consistency. (main): Start 1 traversal and N scanner threads if needed. 2019-01-02 Mark Wielaard * debuginfod.cxx (default_connect_timeout): Removed. (default_transfer_timeout): Removed. (default_timeout): New. Default to 90 seconds. (debuginfod_query_server): Parse server_timeout_envvar as one number. Set as CURLOPT_LOW_SPEED_TIME, with CURL_OPT_LOW_SPEED_LIMITE as 100K. 2020-01-09 Frank Ch. Eigler * debuginfod-client.c (add_extra_headers): New function, based on mjw's draft. (debuginfod_query_server): Call it. 2019-12-22 Frank Ch. Eigler * debuginfod.cxx (*_rpm_*): Rename to *_archive_* throughout. (scan_archives): New read-mostly global to identify archive file extensions and corresponding extractor commands. (parse_opt): Handle new -U flag. 2019-12-19 Frank Ch. Eigler * debuginfod-client.c (default_progressfn): New function. (debuginfod_begin): Use it if $DEBUGINFOD_PROGRESS set. (server_timeout): Bump to 30 seconds. (debuginfod_query_server): Call progressfn -after- rather than before curl ops, to make it likely that a successful transfer results in final a=b call. Tweak cleanup sequence. * debuginfod.h: Document $DEBUGINFOD_PROGRESS name. 2019-12-09 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Check server_urls_envvar early. 2019-12-03 Mark Wielaard * debuginfod-client.c (debuginfod_query_server): Use separate local variables for CURLcode curl_res and CURLMcode curlm_res. 2019-11-26 Mark Wielaard * Makefile.am (BUILD_STATIC): Add needed libraries for libdw and libdebuginfod. 2019-11-25 Frank Ch. Eigler * debuginfod.cxx (groom): Add a sqlite3_db_release_memory() at the end of periodic grooming to try to shrink the process. 2019-11-24 Mark Wielaard * debuginfod.cxx (test_webapi_sleep): Removed. (handler_cb): Don't check test_webapi_sleep and sleep. (main): Don't set test_webapi_sleep. 2019-11-24 Mark Wielaard * debuginfod.cxx (add_metric): New function. (scan_source_file_path): Record metrics for found_executable_total, found_debuginfo_total and found_sourcerefs_total. (scan_source_rpm_path): Likewise. 2019-11-07 Frank Ch. Eigler * debuginfod.cxx: Add /metrics endpoint. Add numerous calls to new functions inc_metric/set_metric to populate threadsafe map containing stats. Add http content-type response headers throughout. (thread_main_*): Simplify counter/timer flow. (main): Reorder web service shutdown to leave http running as long as possible. * debuginfod.8: Document it, add security caution. 2019-11-06 Frank Ch. Eigler * debuginfod.cxx: Add new -L (symlink-following) mode. * debuginfod.8: Document it. 2019-11-04 Frank Ch. Eigler * debuginfo-client.c (debuginfod_set_progressfn): New function for progress/interrupt callback. (debuginfod_clean_cache, debuginfod_query_server): Call it. * debuginfo.h: Declare it. * debuginfod_set_progressfn.3, *_find_debuginfo.3: Document it. * Makefile.am: Install it. * libdebuginfod.map: Export it all under ELFUTILS_0.178 symversion. * debuginfod-find.c: Add -v option to activate progress cb. * debuginfod-find.1: Document it. * debuginfod.cxx: Add $DEBUGINFOD_TEST_WEBAPI_SLEEP env var to insert sleep in webapi callbacks, to help manual testing. 2019-10-28 Frank Ch. Eigler * debuginfod.cxx: New file: debuginfod server. * debuginfod.8: New file: man page. * Makefile.am: Build it. 2019-10-28 Aaron Merey * debuginfod-client.c: New file: debuginfod client library. * debuginfod.h: New file: header for same. * libdebuginfod.map: New file: govern its solib exports. * debuginfod-find.c: New file: command line frontend. * debuginfod-find.1, debuginfod_find_source.3, debuginfod_find_executable.3, debuginfod_find_debuginfo.3: New man pages.