summaryrefslogtreecommitdiff
path: root/tests/pack
Commit message (Collapse)AuthorAgeFilesLines
* tests: declare functions statically where appropriateEdward Thomson2021-11-111-1/+1
|
* path: separate git-specific path functions from utilEdward Thomson2021-11-092-3/+3
| | | | | | Introduce `git_fs_path`, which operates on generic filesystem paths. `git_path` will be kept for only git-specific path functionality (for example, checking for `.git` in a path).
* str: introduce `git_str` for internal, `git_buf` is externalethomson/gitstrEdward Thomson2021-10-174-52/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | libgit2 has two distinct requirements that were previously solved by `git_buf`. We require: 1. A general purpose string class that provides a number of utility APIs for manipulating data (eg, concatenating, truncating, etc). 2. A structure that we can use to return strings to callers that they can take ownership of. By using a single class (`git_buf`) for both of these purposes, we have confused the API to the point that refactorings are difficult and reasoning about correctness is also difficult. Move the utility class `git_buf` to be called `git_str`: this represents its general purpose, as an internal string buffer class. The name also is an homage to Junio Hamano ("gitstr"). The public API remains `git_buf`, and has a much smaller footprint. It is generally only used as an "out" param with strict requirements that follow the documentation. (Exceptions exist for some legacy APIs to avoid breaking callers unnecessarily.) Utility functions exist to convert a user-specified `git_buf` to a `git_str` so that we can call internal functions, then converting it back again.
* hash: hash functions operate on byte arrays not git_oidsEdward Thomson2021-10-021-1/+1
| | | | | | Separate the concerns of the hash functions from the git_oid functions. The git_oid structure will need to understand either SHA1 or SHA256; the hash functions should only deal with the appropriate one of these.
* hash: accept the algorithm in inputsEdward Thomson2021-10-011-1/+1
|
* Merge branch 'main' into multi-pack-index-odb-writeEdward Thomson2021-08-291-1/+1
|\
| * midx: Add a way to write multi-pack-index fileslhchavez2021-07-261-0/+30
| | | | | | | | | | | | | | This change adds the git_midx_writer_* functions to allow to write and create `multi-pack-index` files from `.idx`/`.pack` files. Part of: #5399
* | midx: Introduce git_odb_write_multi_pack_index()lhchavez2021-08-271-0/+34
| | | | | | | | | | | | | | | | This change introduces git_odb_write_multi_pack_index(), which creates a `multi-pack-index` file from all the `.pack` files that have been loaded in the ODB. Fixes: #5399
* | midx: Add a way to write multi-pack-index fileslhchavez2021-08-271-0/+30
|/ | | | | | | This change adds the git_midx_writer_* functions to allow to write and create `multi-pack-index` files from `.idx`/`.pack` files. Part of: #5399
* midx: Fix a bug in `git_midx_needs_refresh()`lhchavez2021-01-071-0/+1
| | | | | The very last check in the freshness check for the midx was wrong >< This was also because this function was not tested.
* tests: ifdef out unused function in no-thread buildsEdward Thomson2020-12-051-0/+2
|
* Make the pack and mwindow implementations data-race-freelhchavez2020-11-282-0/+61
| | | | | | | | | | | | | | | | | | | | | | | | | This change fixes a packfile heap corruption that can happen when interacting with multiple packfiles concurrently across multiple threads. This is exacerbated by setting a lower mwindow open file limit. This change: * Renames most of the internal methods in pack.c to clearly indicate that they expect to be called with a certain lock held, making reasoning about the state of locks a bit easier. * Splits the `git_pack_file` lock in two: the one in `git_pack_file` only protects the `index_map`. The protection to `git_mwindow_file` is now in that struct. * Explicitly checks for freshness of the `git_pack_file` in `git_packfile_unpack_header`: this allows the mwindow implementation to close files whenever there is enough cache pressure, and `git_packfile_unpack_header` will reopen the packfile if needed. * After a call to `p_munmap()`, the `data` and `len` fields are poisoned with `NULL` to make use-after-frees more evident and crash rather than being open to the possibility of heap corruption. * Adds a test case to prevent this from regressing in the future. Fixes: #5591
* midx: Support multi-pack-index files in odb_pack.clhchavez2020-11-271-0/+16
| | | | | | | | | | | | This change adds support for reading multi-pack-index files from the packfile odb backend. This also makes git_pack_file objects open their backing failes lazily in more scenarios, since the multi-pack-index can avoid having to open them in some cases (yay!). This change also refreshes the documentation found in src/odb_pack.c to match the updated code. Part of: #5399
* runtime: move init/shutdown into the "runtime"Edward Thomson2020-10-111-1/+0
| | | | | Provide a mechanism for system components to register for initialization and shutdown of the libgit2 runtime.
* multipack: Introduce a parser for multi-pack-index fileslhchavez2020-10-051-0/+29
| | | | | | | This change is the first in a series to add support for git's multi-pack-index. This should speed up large repositories significantly. Part of: #5399
* Minor nits and style formattinglhchavez2020-07-121-17/+15
|
* Create the repository within the testlhchavez2020-06-271-23/+89
| | | | | This change moves the humongous static repository with 1025 commits into the test file, now with a more modest 16 commits.
* Review feedbacklhchavez2020-06-261-1/+37
| | | | | | | | | | * Change the default of the file limit to 0 (unlimited). * Changed the heuristic to close files to be the file that contains the least-recently-used window such that the window is the most-recently-used in the file, and the file does not have in-use windows. * Parameterized the filelimit test to check for a limit of 1 and 100 open windows.
* mwindow: set limit on number of open fileslhchavez2020-06-211-0/+36
| | | | | | | | | | | | | | | | | | There are some cases in which repositories accrue a large number of packfiles. The existing mwindow limit applies only to the total size of mmap'd files, not on their number. This leads to a situation in which having lots of small packfiles could exhaust the allowed number of open files, particularly on macOS, where the default ulimit is very low (256). This change adds a new configuration parameter (GIT_OPT_SET_MWINDOW_FILE_LIMIT) that sets the maximum number of open packfiles, with a default of 128. This is low enough so that even macOS users should not hit it during normal use. Based on PR #5386, originally written by @josharian. Fixes: #2758
* git_packbuilder_write: Allow setting path to NULL to use the default pathJosh Triplett2020-05-231-0/+9
| | | | | | If given a NULL path, write to the object path of the repository. Add tests for the new behavior.
* tests: pack: add missing asserts around `git_packbuilder_write`Patrick Steinhardt2020-01-091-5/+5
|
* fileops: rename to "futils.h" to match function signaturesPatrick Steinhardt2019-07-202-2/+2
| | | | | | | | | Our file utils functions all have a "futils" prefix, e.g. `git_futils_touch`. One would thus naturally guess that their definitions and implementation would live in files "futils.h" and "futils.c", respectively, but in fact they live in "fileops.h". Rename the files to match expectations.
* indexer: use git_indexer_progress throughoutEdward Thomson2019-02-222-10/+10
| | | | | Update internal usage of `git_transfer_progress` to `git_indexer_progreses`.
* maps: provide high-level iteration interfacePatrick Steinhardt2019-02-151-1/+1
| | | | | | | | | | | | | | | Currently, our headers need to leak some implementation details of maps due to their direct use of indices in the implementation of their foreach macros. This makes it impossible to completely hide the map structures away, and also makes it impossible to include the khash implementation header in the C files of the respective map only. This is now being fixed by providing a high-level iteration interface `map_iterate`, which takes as inputs the map that shall be iterated over, an iterator as well as the locations where keys and values shall be put into. For simplicity's sake, the iterator is a simple `size_t` that shall initialized to `0` on the first call. All existing foreach macros are then adjusted to make use of this new function.
* maps: provide a uniform entry count interfacePatrick Steinhardt2019-02-151-2/+2
| | | | | | | | | There currently exist two different function names for getting the entry count of maps, where offmaps offset and string maps use `num_entries` and OID maps use `size`. In most programming languages with built-in map types, this is simply called `size`, which is also shorter to type. Thus, this commit renames the other two functions `num_entries` to match the common way and adjusts all callers.
* Allow bypassing check '.keep' files using libgit2 option ↵Dhruva Krishnamurthy2019-02-021-0/+10
| | | | 'GIT_OPT_IGNORE_PACK_KEEP_FILE_CHECK'
* git_error: use new names in internal APIs and usageEdward Thomson2019-01-221-6/+6
| | | | | Move to the `git_error` name in the internal API for error-related functions.
* object_type: use new enumeration namesethomson/index_fixesEdward Thomson2018-12-013-5/+5
| | | | Use the new object_type enumeration names within the codebase.
* khash: remove intricate knowledge of khash typesPatrick Steinhardt2018-11-281-1/+1
| | | | | | | Instead of using the `khiter_t`, `git_strmap_iter` and `khint_t` types, simply use `size_t` instead. This decouples code from the khash stuff and makes it possible to move the khash includes into the implementation files.
* tests: indexer: add test to exercise our connectivity checkingPatrick Steinhardt2018-06-221-0/+58
| | | | | | The new connectivity tests are not currently being verified at all due to being turned off by default. Create two test cases for a pack file which fails our checks and one which suceeds.
* indexer: introduce options struct to `git_indexer_new`Patrick Steinhardt2018-06-222-10/+10
| | | | | | | | | | We strive to keep an options structure to many functions to be able to extend options in the future without breaking the API. `git_indexer_new` doesn't have one right now, but we want to be able to add an option for enabling strict packfile verification. Add a new `git_indexer_options` structure and adjust callers to use that.
* Convert usage of `git_buf_free` to new `git_buf_dispose`Patrick Steinhardt2018-06-102-5/+5
|
* Fix unpack double freelhchavez2017-12-231-5/+56
| | | | | | | | | | | If an element has been cached, but then the call to packfile_unpack_compressed() fails, the very next thing that happens is that its data is freed and then the element is not removed from the cache, which frees the data again. This change sets obj->data to NULL to avoid the double-free. It also stops trying to resolve deltas after two continuous failed rounds of resolution, and adds a test for this.
* libFuzzer: Fix missing trailer crashlhchavez2017-12-081-0/+17
| | | | | | | This change fixes an invalid memory access when the trailer is missing / corrupt. Found using libFuzzer.
* libFuzzer: Fix a git_packfile_stream leaklhchavez2017-12-061-0/+27
| | | | | | | This change ensures that the git_packfile_stream object in git_indexer_append() does not leak when the stream has errors. Found using libFuzzer.
* Merge pull request #4088 from chescock/packfile-name-using-complete-hashEdward Thomson2017-06-112-6/+6
|\ | | | | Ensure packfiles with different contents have different names
| * indexer: name pack files after trailer hashChris Hescock2017-05-192-6/+6
| | | | | | | | | | | | | | | | | | | | | | Upstream git.git has changed the way how packfiles are named. Previously, they were using a hash of the contained object's OIDs, which has then been changed to use the hash of the complete packfile instead. See 1190a1acf (pack-objects: name pack files after trailer hash, 2013-12-05) in the git.git repository for more information on this change. This commit changes our logic to match the behavior of core git.
* | settings: rename `GIT_OPT_ENABLE_SYNCHRONOUS_OBJECT_CREATION`Patrick Steinhardt2017-06-081-2/+2
| | | | | | | | | | | | | | | | | | | | | | Initially, the setting has been solely used to enable the use of `fsync()` when creating objects. Since then, the use has been extended to also cover references and index files. As the option is not yet part of any release, we can still correct this by renaming the option to something more sensible, indicating not only correlation to objects. This commit renames the option to `GIT_OPT_ENABLE_FSYNC_GITDIR`. We also move the variable from the object to repository source code.
* | Honor `core.fsyncObjectFiles`ethomson/fsyncEdward Thomson2017-03-021-8/+17
| |
* | win32: don't fsync parent directories on WindowsEdward Thomson2017-02-281-1/+10
| | | | | | | | Windows doesn't support it.
* | fsync: call it "synchronous" object writingEdward Thomson2017-02-281-2/+2
| | | | | | | | | | Rename `GIT_OPT_ENABLE_SYNCHRONIZED_OBJECT_CREATION` -> `GIT_OPT_ENABLE_SYNCHRONOUS_OBJECT_CREATION`.
* | fsync parent directories when fsyncingEdward Thomson2017-02-281-1/+1
| | | | | | | | | | | | When fsync'ing files, fsync the parent directory in the case where we rename a file into place, or create a new file, to ensure that the directory entry is flushed correctly.
* | packbuilder: honor git_object__synchronized_writingEdward Thomson2017-02-281-0/+19
|/ | | | | Honor `git_object__synchronized_writing` when creating a packfile and corresponding index.
* Addressed review feedbacklhchavez2017-01-141-10/+3
|
* Fix the memory leaklhchavez2017-01-031-1/+9
|
* Add a testlhchavez2017-01-011-0/+40
|
* Silence unused return value warningJacques Germishuys2014-09-261-1/+1
|
* Factor 40 and 41 constants from source.Ciro Santilli2014-09-161-2/+2
|
* Introduce cl_assert_equal_oidEdward Thomson2014-07-011-2/+2
|
* Share packs across repository instancescmn/global-mwfCarlos Martín Nieto2014-06-231-0/+42
| | | | | | | | | | | Opening the same repository multiple times will currently open the same file multiple times, as well as map the same region of the file multiple times. This is not necessary, as the packfile data is immutable. Instead of opening and closing packfiles directly, introduce an indirection and allocate packfiles globally. This does mean locking on each packfile open, but we already use this lock for the global mwindow list so it doesn't introduce a new contention point.
View Lorry Controller Status