summaryrefslogtreecommitdiff
path: root/src/util/fossilize_db.c
Commit message (Collapse)AuthorAgeFilesLines
* util/fossilize_db: add ifdef for inotify headerJuston Li2023-01-261-1/+4
| | | | | | | | | | FOZ_DB_UTIL_DYNAMIC_LIST is defined if the inotify header was detected. Fixes: 3b69b67545b ("util/fossilize_db: add runtime RO foz db loading via FOZ_DBS_DYNAMIC_LIST") Signed-off-by: Juston Li <justonli@google.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20865>
* util/fossilize_db: fix macOS inotify build errorJuston Li2023-01-231-0/+6
| | | | | | | | | require <sys/inotify.h> for fossilize_db Fixes: 3b69b67545b ("util/fossilize_db: add runtime RO foz db loading via FOZ_DBS_DYNAMIC_LIST") Signed-off-by: Juston Li <justonli@google.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20755>
* util/fossilize_db: add runtime RO foz db loading via FOZ_DBS_DYNAMIC_LISTJuston Li2023-01-161-6/+194
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a new environment varible MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMICE_LIST that specifies a text file containing a list of RO fossilize caches to load. The list file is modifiable at runtime to allow for loading RO caches after initialization unlike MESA_DISK_CACHE_READ_ONLY_FOZ_DBS. The implementation spawns an updater thread that uses inotify to monitor the list file for modifications, attempting to load new foz dbs added to the list. Removing files from the list will not evict a loaded cache. MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMIC_LIST takes an absolute path. The file must exist at initialization for updating to occur. File names of foz dbs in the list file are new-line separated and take relative paths to the default cache directory like MESA_DISK_CACHE_READ_ONLY_FOZ_DBS. The maximum number of RO foz dbs is kept to 8 and is shared between MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMIC_LIST and MESA_DISK_CACHE_READ_ONLY_FOZ_DBS. The intended use case for this feature is to allow prebuilt caches to be downloaded and loaded asynchronously during app runtime. Prebuilt caches be large (several GB) and depending on network conditions would otherwise present extended wait time for caches to be availible before app launch. This will be used in Chrome OS. Signed-off-by: Juston Li <justonli@google.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19328>
* utils/fossilize_db: refactor out loading RO foz dbsJuston Li2023-01-161-40/+47
| | | | | | | Move loading of RO foz dbs into its own load_ro_foz_dbs() function Signed-off-by: Juston Li <justonli@google.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19328>
* util/fossilize_db: don't destroy foz on RO load failJuston Li2023-01-161-2/+4
| | | | | | | | | | | When loading multiple RO foz dbs, if a db fails to load, continue trying to load other RO foz dbs instead of destroying the foz cache. Preserve destroying the foz cache and not preceding to load RO caches if the RW cache fails to load. Signed-off-by: Juston Li <justonli@google.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19328>
* util/disk_cache: Support combined foz ro and non-foz rw cachesDmitry Osipenko2023-01-091-16/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Mesa utilizes only one type of cache at a time. This patch enables support for combined reading from read-only Fossilize cache + non-foz read-write caches. From now on, a non-foz read-write caches will first try to retrieve data from a read-only foz cache if new MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ environment variable is set to true, otherwise the caching behaviour is unchanged. The new flag has no effect when MESA_DISK_CACHE_SINGLE_FILE=1, i.e. when the single-file foz cache is used. This change allows us to ship a prebuilt RO caches for a certain applications, while the rest of applications will benefit from the regular RW caching that supports cache-size limitation. This feature will be used by ChromeOS. Usage example #1: MESA_DISK_CACHE_DATABASE=0 MESA_DISK_CACHE_SINGLE_FILE=0 MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ=1 MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=rocache1,rocache2 Usage example #2: MESA_DISK_CACHE_DATABASE=1 MESA_DISK_CACHE_SINGLE_FILE=0 MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ=1 MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=rocache1,rocache2 Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18551>
* util/fossilize_db: Fix resource leaks in foz_prepare() error pathsDmitry Osipenko2023-01-091-4/+11
| | | | | | | | | | | | The foz_prepare() doesn't perform cleanup on failure and then foz_destroy() is never invoked for the foz_db, causing minor memory and FD leaks. Add the cleanup to foz_prepare() error code paths. Make foz_destroy() to clear the foz_db struct for consistency, right now the destroying is invoked only once, but Mesa cache error code paths aren't trivial and may change in the future. Suggested-by: Timothy Arceri <tarceri@itsqueeze.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18551>
* util/fossilize_db: Fix double free in error handling.Bas Nieuwenhuizen2022-01-241-1/+5
| | | | | | | | If the file ptr is not NULL then foz_destroy will also try to destroy it. Fixes: eca6bb9540d ("util/fossilize_db: add basic fossilize db util to read/write shader caches") Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14671>
* util/fossilize_db: Don't corrupt keys during entry read.Bas Nieuwenhuizen2021-09-041-6/+6
| | | | | | | | We change the hash that is still used to set entry->key. Fixes: d2d642cc014 "util/fossilize_db: Only allocate entries after full read." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12725>
* util/fossilize_db: Add extra flock mutex.Bas Nieuwenhuizen2021-08-081-1/+11
| | | | | | | | | | | | The flock is per-fd, not per thread, and we do it outside of the main mutex. This was done to avoid having to wait in the mutex, but we can get a case where one ends up running the body with the flock unlocked. Fix this by adding a mutex that doesn't need to be locked for reads. Fixes: 4f0f8133a35 "util/fossilize_db: Do not lock the fossilize db permanently." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12266>
* util/fossilize_db: Unlock the cache file if the entry already exists.Bas Nieuwenhuizen2021-08-081-0/+1
| | | | | | Fixes: 4f0f8133a35 "util/fossilize_db: Do not lock the fossilize db permanently." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12266>
* util/fossilize_db: Use uint64_t for file size.Bas Nieuwenhuizen2021-08-081-1/+1
| | | | | | | | For those 32-bit systems with 4G of cache. Fixes: 2ec1bff0f3a "util/fossilize_db: Split out reading the index." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12266>
* util/fossilize_db: Only allocate entries after full read.Bas Nieuwenhuizen2021-08-071-10/+11
| | | | | | | | Should void leaking entries on read failure. Fixes: 2ec1bff0f3a "util/fossilize_db: Split out reading the index." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12204>
* util/fossilize_db: Be conservative about header length check for locking.Bas Nieuwenhuizen2021-08-071-2/+3
| | | | | | | | | | Don't anticipate seeing any partial written headers but just in case we should probably wait on the lock to make sure whatever header was being written is finished being written. Fixes: 4f0f8133a35 "util/fossilize_db: Do not lock the fossilize db permanently." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12204>
* util/fossilize_db: Flush files after header write.Bas Nieuwenhuizen2021-08-071-0/+3
| | | | | | | | We should probably flush before we unlock the file again. Fixes: 4f0f8133a35 "util/fossilize_db: Do not lock the fossilize db permanently." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12204>
* util/fossilize_db: Reset file position to parsed_offset on cache_offset read ↵Bas Nieuwenhuizen2021-08-071-1/+1
| | | | | | | | | | failure. Otherwise we might restart reading from the middle of the entry. Fixes: 2ec1bff0f3a "util/fossilize_db: Split out reading the index." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12204>
* util/fossilize_db: Update parsed_offset correctly.Bas Nieuwenhuizen2021-08-071-2/+1
| | | | | | | | | | If things went perfectly parsed_offset was never updated for the final entry and we'd seek_set to the start of the last entry. Is fun when appending to the file next. Fixes: 2ec1bff0f3a "util/fossilize_db: Split out reading the index." Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12204>
* util/foz: stop crashing on destroy if prepare hasn't been calledMike Blumenkrantz2021-07-171-1/+2
| | | | | | | Fixes: eca6bb9540d ("util/fossilize_db: add basic fossilize db util to read/write shader caches") Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11931>
* util/fossilize_db: Only lock the db file, not the index.Bas Nieuwenhuizen2021-06-211-12/+0
| | | | | | | | | | This thing is entirely opt-in wrt caring about it when writing to a file anyway. Since we also lock the two at the same time and they have an 1-1 relation we can just lock one of the two files. Saves some syscalls. Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11485>
* util/fossilize_db: Do not lock the fossilize db permanently.Bas Nieuwenhuizen2021-06-211-15/+62
| | | | | | | | | | | | | | | | This avoids all locks for reads and using lock only while actually writing. This is enabled by doing two things: 1) Reading the index incrementally. This way we get new entries written by other processes and do not write duplicate entries. 2) Taking the lock only during writes, and applying the incremental read while holding the lock so we always append to the actual end of the file. Fixes: eca6bb9540d ("util/fossilize_db: add basic fossilize db util to read/write shader caches") Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11485>
* util/fossilize_db: Split out reading the index.Bas Nieuwenhuizen2021-06-211-61/+75
| | | | | | Fixes: eca6bb9540d ("util/fossilize_db: add basic fossilize db util to read/write shader caches") Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11485>
* util/fossilize_db: Pull seek into lock.Bas Nieuwenhuizen2021-06-211-3/+3
| | | | | | | | Otherwise the seek can overlap with other reads/writes. Fixes: eca6bb9540d ("util/fossilize_db: add basic fossilize db util to read/write shader caches") Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11485>
* util/hash_table: do not leak u64 struct keyJuan A. Suarez Romero2021-04-291-1/+1
| | | | | | | | | | | | | | | For non 64bit devices the key stored in hash_table_u64 is wrapped in hash_key_u64 structure, which is never free. This commit fixes this issue by just removing the user-defined `delete_function` parameter in hash_table_u64_{destroy,clear} (which nobody is using) and using instead a delete function to free this structure. Fixes: 608257cf82f ("i965: Fix INTEL_DEBUG=bat") Reviewed-by: Ian Romanick <ian.d.romanick@intel.com> Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10480>
* util/fossilize_db: remove compression from foz db helperTimothy Arceri2021-03-151-55/+14
| | | | | | | | | We now handle compression in the shared cache item creation code. Compressing the cache item header with the already compressed blob doesn't help much so lets just remove it. Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9593>
* util/disk_cache: make MESA_DISK_CACHE_READ_ONLY_FOZ_DBS a relative pathTimothy Arceri2021-03-041-6/+7
| | | | | | | Rather than passing in full paths this changes things so that we can just pass in filenames relative to the current cache directory. Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9279>
* util/disk_cache: fix crash in fossilize_dbTimothy Arceri2021-02-251-3/+1
| | | | | | | | | | | We don't need to close the file here check_files_opened_successfully() already does that for us if needed. This fixes a crash when invalid filename/paths are provided to the MESA_DISK_CACHE_READ_ONLY_FOZ_DBS environment variable. Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9251>
* util/fossilize_db: Fix compile error with clangRob Clark2021-02-221-1/+1
| | | | | | | | | | ../src/util/fossilize_db.c:178:58: error: use of GNU empty initializer extension [-Werror,-Wgnu-empty-initializer] char hash_str[FOSSILIZE_BLOB_HASH_LENGTH + 1] = {}; Fixes: eca6bb9540d ("util/fossilize_db: add basic fossilize db util to read/write shader caches") Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9180>
* util/fossilize_db: add basic fossilize db util to read/write shader cachesTimothy Arceri2021-02-211-0/+540
My benchmarking shows no significant change in cache load times with a single shader cache file vs the existing cache implementation of many small files (tested with my spinning rust HDD). However this new single file cache implementation does reduce the total size on disk used by the shader cache. We have a problem with the existing cache where writing tiny files to disk causes more disk space to be used than is actually needed for the files due to the minimum size required for a file. In pratice this tends to inflate the size of the cache on disk to over 3x larger. There are other advantages of using a single file for shader cache entries such as allowing better removal of cache entries once we hit the max cache size limit (although we don't implement any max cache size handling in this initial implementation). The primary reason for implementing a single file cache for now is to allow better performance and handling by third party applications such as steam that collect and distribute precompiled cache entries. For this reason we also implement a new environment variable MESA_DISK_CACHE_READ_ONLY_FOZ_DBS which allows a user to pass in a path to a number of external read only shader cache dbs. There is an initial limit of 8 dbs that can be passed to mesa like so: MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=/full_path/filename1, ... ,/full_path/filename8 Where the filename represents the cache db and its index file e.g. filename1.foz and filename1_idx.foz Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
View Lorry Controller Status