summaryrefslogtreecommitdiff
path: root/read-cache.c
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'en/header-split-cache-h-part-2'Junio C Hamano2023-05-091-68/+13
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | More header clean-up. * en/header-split-cache-h-part-2: (22 commits) reftable: ensure git-compat-util.h is the first (indirect) include diff.h: reduce unnecessary includes object-store.h: reduce unnecessary includes commit.h: reduce unnecessary includes fsmonitor: reduce includes of cache.h cache.h: remove unnecessary headers treewide: remove cache.h inclusion due to previous changes cache,tree: move basic name compare functions from read-cache to tree cache,tree: move cmp_cache_name_compare from tree.[ch] to read-cache.c hash-ll.h: split out of hash.h to remove dependency on repository.h tree-diff.c: move S_DIFFTREE_IFXMIN_NEQ define from cache.h dir.h: move DTYPE defines from cache.h versioncmp.h: move declarations for versioncmp.c functions from cache.h ws.h: move declarations for ws.c functions from cache.h match-trees.h: move declarations for match-trees.c functions from cache.h pkt-line.h: move declarations for pkt-line.c functions from cache.h base85.h: move declarations for base85.c functions from cache.h copy.h: move declarations for copy.c functions from cache.h server-info.h: move declarations for server-info.c functions from cache.h packfile.h: move pack_window and pack_entry from cache.h ...
| * commit.h: reduce unnecessary includesElijah Newren2023-04-241-0/+1
| | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * cache,tree: move basic name compare functions from read-cache to treeElijah Newren2023-04-241-68/+0
| | | | | | | | | | | | | | | | | | | | None of base_name_compare(), df_name_compare(), or name_compare() depended upon a cache_entry or index_state in any way. By moving these functions to tree.h, half a dozen other files can stop depending upon cache.h (though that change will be made in a later commit). Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * cache,tree: move cmp_cache_name_compare from tree.[ch] to read-cache.cElijah Newren2023-04-241-1/+12
| | | | | | | | | | | | | | | | | | | | Since cmp_cache_name_compare() was comparing cache_entry structs, it was associated with the cache rather than with trees. Move the function. As a side effect, we can make cache_name_stage_compare() static as well. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * symlinks.h: move declarations for symlinks.c functions from cache.hElijah Newren2023-04-241-0/+1
| | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'en/header-split-cache-h'Junio C Hamano2023-04-251-0/+5
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Header clean-up. * en/header-split-cache-h: (24 commits) protocol.h: move definition of DEFAULT_GIT_PORT from cache.h mailmap, quote: move declarations of global vars to correct unit treewide: reduce includes of cache.h in other headers treewide: remove double forward declaration of read_in_full cache.h: remove unnecessary includes treewide: remove cache.h inclusion due to pager.h changes pager.h: move declarations for pager.c functions from cache.h treewide: remove cache.h inclusion due to editor.h changes editor: move editor-related functions and declarations into common file treewide: remove cache.h inclusion due to object.h changes object.h: move some inline functions and defines from cache.h treewide: remove cache.h inclusion due to object-file.h changes object-file.h: move declarations for object-file.c functions from cache.h treewide: remove cache.h inclusion due to git-zlib changes git-zlib: move declarations for git-zlib functions from cache.h treewide: remove cache.h inclusion due to object-name.h changes object-name.h: move declarations for object-name.c functions from cache.h treewide: remove unnecessary cache.h inclusion treewide: be explicit about dependence on mem-pool.h treewide: be explicit about dependence on oid-array.h ...
| * object-file.h: move declarations for object-file.c functions from cache.hElijah Newren2023-04-111-0/+1
| | | | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Acked-by: Calvin Wan <calvinwan@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * object-name.h: move declarations for object-name.c functions from cache.hElijah Newren2023-04-111-0/+1
| | | | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Acked-by: Calvin Wan <calvinwan@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * treewide: be explicit about dependence on mem-pool.hElijah Newren2023-04-111-0/+1
| | | | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Acked-by: Calvin Wan <calvinwan@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * treewide: be explicit about dependence on oid-array.hElijah Newren2023-04-111-0/+1
| | | | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Acked-by: Calvin Wan <calvinwan@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * treewide: be explicit about dependence on trace.h & trace2.hElijah Newren2023-04-111-0/+1
| | | | | | | | | | | | | | | | | | | | | | Dozens of files made use of trace and trace2 functions, without explicitly including trace.h or trace2.h. This made it more difficult to find which files could remove a dependence on cache.h. Make C files explicitly include trace.h or trace2.h if they are using them. Signed-off-by: Elijah Newren <newren@gmail.com> Acked-by: Calvin Wan <calvinwan@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * Merge branch 'ab/remove-implicit-use-of-the-repository' into ↵Junio C Hamano2023-04-041-4/+5
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | en/header-split-cache-h * ab/remove-implicit-use-of-the-repository: libs: use "struct repository *" argument, not "the_repository" post-cocci: adjust comments for recent repo_* migration cocci: apply the "revision.h" part of "the_repository.pending" cocci: apply the "rerere.h" part of "the_repository.pending" cocci: apply the "refs.h" part of "the_repository.pending" cocci: apply the "promisor-remote.h" part of "the_repository.pending" cocci: apply the "packfile.h" part of "the_repository.pending" cocci: apply the "pretty.h" part of "the_repository.pending" cocci: apply the "object-store.h" part of "the_repository.pending" cocci: apply the "diff.h" part of "the_repository.pending" cocci: apply the "commit.h" part of "the_repository.pending" cocci: apply the "commit-reach.h" part of "the_repository.pending" cocci: apply the "cache.h" part of "the_repository.pending" cocci: add missing "the_repository" macros to "pending" cocci: sort "the_repository" rules by header cocci: fix incorrect & verbose "the_repository" rules cocci: remove dead rule from "the_repository.pending.cocci"
* | \ Merge branch 'en/header-split-cleanup'Junio C Hamano2023-04-061-0/+3
|\ \ \ | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Split key function and data structure definitions out of cache.h to new header files and adjust the users. * en/header-split-cleanup: csum-file.h: remove unnecessary inclusion of cache.h write-or-die.h: move declarations for write-or-die.c functions from cache.h treewide: remove cache.h inclusion due to setup.h changes setup.h: move declarations for setup.c functions from cache.h treewide: remove cache.h inclusion due to environment.h changes environment.h: move declarations for environment.c functions from cache.h treewide: remove unnecessary includes of cache.h wrapper.h: move declarations for wrapper.c functions from cache.h path.h: move function declarations for path.c functions from cache.h cache.h: remove expand_user_path() abspath.h: move absolute path functions from cache.h environment: move comment_line_char from cache.h treewide: remove unnecessary cache.h inclusion from several sources treewide: remove unnecessary inclusion of gettext.h treewide: be explicit about dependence on gettext.h treewide: remove unnecessary cache.h inclusion from a few headers
| * | environment.h: move declarations for environment.c functions from cache.hElijah Newren2023-03-211-0/+1
| | | | | | | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | wrapper.h: move declarations for wrapper.c functions from cache.hElijah Newren2023-03-211-0/+1
| | | | | | | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | treewide: be explicit about dependence on gettext.hElijah Newren2023-03-211-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Dozens of files made use of gettext functions, without explicitly including gettext.h. This made it more difficult to find which files could remove a dependence on cache.h. Make C files explicitly include gettext.h if they are using it. However, while compat/fsmonitor/fsm-ipc-darwin.c should also gain an include of gettext.h, it was left out to avoid conflicting with an in-flight topic. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | Merge branch 'ab/remove-implicit-use-of-the-repository'Junio C Hamano2023-04-061-4/+5
|\ \ \ | | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Code clean-up around the use of the_repository. * ab/remove-implicit-use-of-the-repository: libs: use "struct repository *" argument, not "the_repository" post-cocci: adjust comments for recent repo_* migration cocci: apply the "revision.h" part of "the_repository.pending" cocci: apply the "rerere.h" part of "the_repository.pending" cocci: apply the "refs.h" part of "the_repository.pending" cocci: apply the "promisor-remote.h" part of "the_repository.pending" cocci: apply the "packfile.h" part of "the_repository.pending" cocci: apply the "pretty.h" part of "the_repository.pending" cocci: apply the "object-store.h" part of "the_repository.pending" cocci: apply the "diff.h" part of "the_repository.pending" cocci: apply the "commit.h" part of "the_repository.pending" cocci: apply the "commit-reach.h" part of "the_repository.pending" cocci: apply the "cache.h" part of "the_repository.pending" cocci: add missing "the_repository" macros to "pending" cocci: sort "the_repository" rules by header cocci: fix incorrect & verbose "the_repository" rules cocci: remove dead rule from "the_repository.pending.cocci"
| * | libs: use "struct repository *" argument, not "the_repository"Ævar Arnfjörð Bjarmason2023-03-281-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As can easily be seen from grepping in our sources, we had these uses of "the_repository" in various library code in cases where the function in question was already getting a "struct repository *" argument. Let's use that argument instead. Out of these changes only the changes to "cache-tree.c", "commit-reach.c", "shallow.c" and "upload-pack.c" would have cleanly applied before the migration away from the "repo_*()" wrapper macros in the preceding commits. The rest aren't new, as we'd previously implicitly refer to "the_repository", but it's now more obvious that we were doing the wrong thing all along, and should have used the parameter instead. The change to change "get_index_format_default(the_repository)" in "read-cache.c" to use the "r" variable instead should arguably have been part of [1], or in the subsequent cleanup in [2]. Let's do it here, as can be seen from the initial code in [3] it's not important that we use "the_repository" there, but would prefer to always use the current repository. This change excludes the "the_repository" use in "upload-pack.c"'s upload_pack_advertise(), as the in-flight [4] makes that change. 1. ee1f0c242ef (read-cache: add index.skipHash config option, 2023-01-06) 2. 6269f8eaad0 (treewide: always have a valid "index_state.repo" member, 2023-01-17) 3. 7211b9e7534 (repo-settings: consolidate some config settings, 2019-08-13) 4. <Y/hbUsGPVNAxTdmS@coredump.intra.peff.net> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | cocci: apply the "object-store.h" part of "the_repository.pending"Ævar Arnfjörð Bjarmason2023-03-281-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | Apply the part of "the_repository.pending.cocci" pertaining to "object-store.h". Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | cocci: apply the "cache.h" part of "the_repository.pending"Ævar Arnfjörð Bjarmason2023-03-281-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Apply the part of "the_repository.pending.cocci" pertaining to "cache.h". Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | Merge branch 'js/split-index-fixes'Junio C Hamano2023-04-041-17/+32
|\ \ \ | |_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The index files can become corrupt under certain conditions when the split-index feature is in use, especially together with fsmonitor, which have been corrected. * js/split-index-fixes: unpack-trees: take care to propagate the split-index flag fsmonitor: avoid overriding `cache_changed` bits split-index; stop abusing the `base_oid` to strip the "link" extension split-index & fsmonitor: demonstrate a bug
| * | split-index; stop abusing the `base_oid` to strip the "link" extensionJohannes Schindelin2023-03-271-17/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a split-index is in effect, the `$GIT_DIR/index` file needs to contain a "link" extension that contains all the information about the split-index, including the information about the shared index. However, in some cases Git needs to suppress writing that "link" extension (i.e. to fall back to writing a full index) even if the in-memory index structure _has_ a `split_index` configured. This is the case e.g. when "too many not shared" index entries exist. In such instances, the current code sets the `base_oid` field of said `split_index` structure to all-zero to indicate that `do_write_index()` should skip writing the "link" extension. This can lead to problems later on, when the in-memory index is still used to perform other operations and eventually wants to write a split-index, detects the presence of the `split_index` and reuses that, too (under the assumption that it has been initialized correctly and still has a non-null `base_oid`). Let's stop zeroing out the `base_oid` to indicate that the "link" extension should not be written. One might be tempted to simply call `discard_split_index()` instead, under the assumption that Git decided to write a non-split index and therefore the `split_index` structure might no longer be wanted. However, that is not possible because that would release index entries in `split_index->base` that are likely to still be in use. Therefore we cannot do that. The next best thing we _can_ do is to introduce a bit field to indicate specifically which index extensions (not) to write. So that's what we do here. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | cache.h: remove dependence on hex.h; make other files include it explicitlyElijah Newren2023-02-231-0/+1
| | | | | | | | | | | | | | | Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | alloc.h: move ALLOC_GROW() functions from cache.hElijah Newren2023-02-231-0/+1
| |/ |/| | | | | | | | | | | | | | | This allows us to replace includes of cache.h with includes of the much smaller alloc.h in many places. It does mean that we also need to add includes of alloc.h in a number of C files. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'rs/size-t-fixes'Junio C Hamano2023-02-151-6/+7
|\ \ | | | | | | | | | | | | | | | | | | | | | Type fixes. * rs/size-t-fixes: pack-objects: use strcspn(3) in name_cmp_len() read-cache: use size_t for {base,df}_name_compare()
| * | read-cache: use size_t for {base,df}_name_compare()René Scharfe2023-02-061-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | Support names of any length in base_name_compare() and df_name_compare() by using size_t for their length parameters. They pass the length on to memcmp(3), which also takes it as a size_t. Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | treewide: always have a valid "index_state.repo" memberÆvar Arnfjörð Bjarmason2023-01-171-12/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the "repo" member was added to "the_index" in [1] the repo_read_index() was made to populate it, but the unpopulated "the_index" variable didn't get the same treatment. Let's do that in initialize_the_repository() when we set it up, and likewise for all of the current callers initialized an empty "struct index_state". This simplifies code that needs to deal with "the_index" or a custom "struct index_state", we no longer need to second-guess this part of the "index_state" deep in the stack. A recent example of such second-guessing is the "istate->repo ? istate->repo : the_repository" code in [2]. We can now simply use "istate->repo". We're doing this by making use of the INDEX_STATE_INIT() macro (and corresponding function) added in [3], which now have mandatory "repo" arguments. Because we now call index_state_init() in repository.c's initialize_the_repository() we don't need to handle the case where we have a "repo->index" whose "repo" member doesn't match the "repo" we're setting up, i.e. the "Complete the double-reference" code in repo_read_index() being altered here. That logic was originally added in [1], and was working around the lack of what we now have in initialize_the_repository(). For "fsmonitor-settings.c" we can remove the initialization of a NULL "r" argument to "the_repository". This was added back in [4], and was needed at the time for callers that would pass us the "r" from an "istate->repo". Before this change such a change to "fsmonitor-settings.c" would segfault all over the test suite (e.g. in t0002-gitfile.sh). This change has wider eventual implications for "fsmonitor-settings.c". The reason the other lazy loading behavior in it is required (starting with "if (!r->settings.fsmonitor) ..." is because of the previously passed "r" being "NULL". I have other local changes on top of this which move its configuration reading to "prepare_repo_settings()" in "repo-settings.c", as we could now start to rely on it being called for our "r". But let's leave all of that for now, and narrowly remove this particular part of the lazy-loading. 1. 1fd9ae517c4 (repository: add repo reference to index_state, 2021-01-23) 2. ee1f0c242ef (read-cache: add index.skipHash config option, 2023-01-06) 3. 2f6b1eb794e (cache API: add a "INDEX_STATE_INIT" macro/function, add release_index(), 2023-01-12) 4. 1e0ea5c4316 (fsmonitor: config settings are repository-specific, 2022-03-25) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Acked-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | Merge branch 'ab/cache-api-cleanup' into ab/cache-api-cleanup-usersJunio C Hamano2023-01-171-19/+24
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * ab/cache-api-cleanup: cache API: add a "INDEX_STATE_INIT" macro/function, add release_index() read-cache.c: refactor set_new_index_sparsity() for subsequent commit sparse-index API: BUG() out on NULL ensure_full_index() sparse-index.c: expand_to_path() can assume non-NULL "istate" builtin/difftool.c: { 0 }-initialize rather than using memset()
| * | | cache API: add a "INDEX_STATE_INIT" macro/function, add release_index()Ævar Arnfjörð Bjarmason2023-01-161-13/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Hopefully in some not so distant future, we'll get advantages from always initializing the "repo" member of the "struct index_state". To make that easier let's introduce an initialization macro & function. The various ad-hoc initialization of the structure can then be changed over to it, and we can remove the various "0" assignments in discard_index() in favor of calling index_state_init() at the end. While not strictly necessary, let's also change the CALLOC_ARRAY() of various "struct index_state *" to use an ALLOC_ARRAY() followed by index_state_init() instead. We're then adding the release_index() function and converting some callers (including some of these allocations) over to it if they either won't need to use their "struct index_state" again, or are just about to call index_state_init(). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Acked-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | | read-cache.c: refactor set_new_index_sparsity() for subsequent commitÆvar Arnfjörð Bjarmason2023-01-131-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactor code added to set_new_index_sparsity() in [1] to eliminate indentation resulting from putting the body of his function within the "if" block. Let's instead return early if we have no istate->repo. This trivial change makes the subsequent commit's diff smaller. 1. 491df5f679f (read-cache: set sparsity when index is new, 2022-05-10) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Acked-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | | Merge branch 'ds/omit-trailing-hash-in-index'Junio C Hamano2023-01-161-1/+13
|\ \ \ \ | |/ / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Introduce an optional configuration to allow the trailing hash that protects the index file from bit flipping. * ds/omit-trailing-hash-in-index: features: feature.manyFiles implies fast index writes test-lib-functions: add helper for trailing hash read-cache: add index.skipHash config option hashfile: allow skipping the hash function
| * | | features: feature.manyFiles implies fast index writesDerrick Stolee2023-01-071-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The recent addition of the index.skipHash config option allows index writes to speed up by skipping the hash computation for the trailing checksum. This is particularly critical for repositories with many files at HEAD, so add this config option to two cases where users in that scenario may opt-in to such behavior: 1. The feature.manyFiles config option enables some options that are helpful for repositories with many files at HEAD. 2. 'scalar register' and 'scalar reconfigure' set config options that optimize for large repositories. In both of these cases, set index.skipHash=true to gain this speedup. Add tests that demonstrate the proper way that index.skipHash=true can override feature.manyFiles=true. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | | read-cache: add index.skipHash config optionDerrick Stolee2023-01-071-1/+12
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The previous change allowed skipping the hashing portion of the hashwrite API, using it instead as a buffered write API. Disabling the hashwrite can be particularly helpful when the write operation is in a critical path. One such critical path is the writing of the index. This operation is so critical that the sparse index was created specifically to reduce the size of the index to make these writes (and reads) faster. This trade-off between file stability at rest and write-time performance is not easy to balance. The index is an interesting case for a couple reasons: 1. Writes block users. Writing the index takes place in many user- blocking foreground operations. The speed improvement directly impacts their use. Other file formats are typically written in the background (commit-graph, multi-pack-index) or are super-critical to correctness (pack-files). 2. Index files are short lived. It is rare that a user leaves an index for a long time with many staged changes. Outside of staged changes, the index can be completely destroyed and rewritten with minimal impact to the user. Following a similar approach to one used in the microsoft/git fork [1], add a new config option (index.skipHash) that allows disabling this hashing during the index write. The cost is that we can no longer validate the contents for corruption-at-rest using the trailing hash. [1] https://github.com/microsoft/git/commit/21fed2d91410f45d85279467f21d717a2db45201 We load this config from the repository config given by istate->repo, with a fallback to the_repository if it is not set. While older Git versions will not recognize the null hash as a special case, the file format itself is still being met in terms of its structure. Using this null hash will still allow Git operations to function across older versions. The one exception is 'git fsck' which checks the hash of the index file. This used to be a check on every index read, but was split out to just the index in a33fc72fe91 (read-cache: force_verify_index_checksum, 2017-04-14) and released first in Git 2.13.0. Document the versions that relaxed these restrictions, with the optimistic expectation that this change will be included in Git 2.40.0. Here, we disable this check if the trailing hash is all zeroes. We add a warning to the config option that this may cause undesirable behavior with older Git versions. As a quick comparison, I tested 'git update-index --force-write' with and without index.skipHash=true on a copy of the Linux kernel repository. Benchmark 1: with hash Time (mean ± σ): 46.3 ms ± 13.8 ms [User: 34.3 ms, System: 11.9 ms] Range (min … max): 34.3 ms … 79.1 ms 82 runs Benchmark 2: without hash Time (mean ± σ): 26.0 ms ± 7.9 ms [User: 11.8 ms, System: 14.2 ms] Range (min … max): 16.3 ms … 42.0 ms 69 runs Summary 'without hash' ran 1.78 ± 0.76 times faster than 'with hash' These performance benefits are substantial enough to allow users the ability to opt-in to this feature, even with the potential confusion with older 'git fsck' versions. Test this new config option, both at a command-line level and within a submodule. The confirmation is currently limited to confirm that 'git fsck' does not complain about the index. Future updates will make this test more robust. It is critical that this test is placed before the test_index_version tests, since those tests obliterate the .git/config file and hence lose the setting from GIT_TEST_DEFAULT_HASH, if set. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | Merge branch 'ab/various-leak-fixes'Junio C Hamano2022-12-141-0/+5
|\ \ \ | |/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Various leak fixes. * ab/various-leak-fixes: built-ins: use free() not UNLEAK() if trivial, rm dead code revert: fix parse_options_concat() leak cherry-pick: free "struct replay_opts" members rebase: don't leak on "--abort" connected.c: free the "struct packed_git" sequencer.c: fix "opts->strategy" leak in read_strategy_opts() ls-files: fix a --with-tree memory leak revision API: call graph_clear() in release_revisions() unpack-file: fix ancient leak in create_temp_file() built-ins & libs & helpers: add/move destructors, fix leaks dir.c: free "ident" and "exclude_per_dir" in "struct untracked_cache" read-cache.c: clear and free "sparse_checkout_patterns" commit: discard partial cache before (re-)reading it {reset,merge}: call discard_index() before returning tests: mark tests as passing with SANITIZE=leak
| * | read-cache.c: clear and free "sparse_checkout_patterns"Ævar Arnfjörð Bjarmason2022-11-211-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The "sparse_checkout_patterns" member was added to the "struct index_state" in 836e25c51b2 (sparse-checkout: hold pattern list in index, 2021-03-30), but wasn't added to discard_index(). Let's do that. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Taylor Blau <me@ttaylorr.com>
* | | read-cache API & users: make discard_index() return voidÆvar Arnfjörð Bjarmason2022-11-211-3/+1
|/ / | | | | | | | | | | | | | | | | | | The discard_index() function has not returned non-zero since 7a51ed66f65 (Make on-disk index representation separate from in-core one, 2008-01-14), but we've had various code in-tree still acting as though that might be the case. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | read-cache: avoid misaligned reads in index v4Victoria Dye2022-09-281-19/+42
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The process for reading the index into memory from disk is to first read its contents into a single memory-mapped file buffer (type 'char *'), then sequentially convert each on-disk index entry into a corresponding incore 'cache_entry'. To access the contents of the on-disk entry for processing, a moving pointer within the memory-mapped file is cast to type 'struct ondisk_cache_entry *'. In index v4, the entries in the on-disk index file are written *without* aligning their first byte to a 4-byte boundary; entries are a variable length (depending on the entry name and whether or not extended flags are used). As a result, casting the 'char *' buffer pointer to 'struct ondisk_cache_entry *' then accessing its contents in a 'SANITIZE=undefined' build can trigger the following error: read-cache.c:1886:46: runtime error: member access within misaligned address <address> for type 'struct ondisk_cache_entry', which requires 4 byte alignment Avoid this error by reading fields directly from the 'char *' buffer, using the 'offsetof' individual fields in 'struct ondisk_cache_entry'. Additionally, add documentation describing why the new approach avoids the misaligned address error, as well as advice on how to improve the implementation in the future. Reported-by: Jeff King <peff@peff.net> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* Merge branch 'vd/sparse-reset-checkout-fixes' into maintJunio C Hamano2022-08-261-0/+5
|\ | | | | | | | | | | | | | | | | | | | | | | Fixes to sparse index compatibility work for "reset" and "checkout" commands. source: <pull.1312.v3.git.1659985672.gitgitgadget@gmail.com> * vd/sparse-reset-checkout-fixes: unpack-trees: unpack new trees as sparse directories cache.h: create 'index_name_pos_sparse()' oneway_diff: handle removed sparse directories checkout: fix nested sparse directory diff in sparse index
| * cache.h: create 'index_name_pos_sparse()'Victoria Dye2022-08-081-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add 'index_name_pos_sparse()', which behaves the same as 'index_name_pos()', except that it does not expand a sparse index to search for an entry inside a sparse directory. 'index_entry_exists()' was originally implemented in 20ec2d034c (reset: make sparse-aware (except --mixed), 2021-11-29) as an alternative to 'index_name_pos()' to allow callers to search for an index entry without expanding a sparse index. However, that particular use case only required knowing whether the requested entry existed, so 'index_entry_exists()' does not return the index positioning information provided by 'index_name_pos()'. This patch implements 'index_name_pos_sparse()' to accommodate callers that need the positioning information of 'index_name_pos()', but do not want to expand the index. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'tk/untracked-cache-with-uall' into maintJunio C Hamano2022-08-101-0/+2
|\ \ | |/ |/| | | | | | | | | | | | | Fix for a bug that makes write-tree to fail to write out a non-existent index as a tree, introduced in 2.37. source: <20220722212232.833188-1-martin.agren@gmail.com> * tk/untracked-cache-with-uall: read-cache: make `do_read_index()` always set up `istate->repo`
| * read-cache: make `do_read_index()` always set up `istate->repo`Martin Ågren2022-07-221-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If there is no index file, e.g., because the repository has just been created, we return zero early (unless `must_exist` makes us die instead.) This early return means we do not set up `istate->repo`. With `core.untrackedCache=true`, the recent e6a653554b ("untracked-cache: support '--untracked-files=all' if configured", 2022-03-31) will eventually pass down `istate->repo` as a null pointer to `repo_config_get_string()`, causing a segmentation fault. If we do hit this early return, set up `istate->repo` similar to when we actually read the index. Reported-by: Joey Hess <id@joeyh.name> Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | read_index_from(): avoid memory leakJohannes Schindelin2022-06-161-3/+3
| | | | | | | | | | | | | | | | | | | | | | In 998330ac2e7c (read-cache: look for shared index files next to the index, too, 2021-08-26), we added code that allocates memory to store the base path of a shared index, but we never released that memory. Reported by Coverity. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'zh/read-cache-copy-name-entry-fix'Junio C Hamano2022-06-131-2/+0
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | Remove redundant copying (with index v3 and older) or possible over-reading beyond end of mmapped memory (with index v4) has been corrected. * zh/read-cache-copy-name-entry-fix: read-cache.c: reduce unnecessary cache entry name copying
| * | read-cache.c: reduce unnecessary cache entry name copyingZheNing Hu2022-06-061-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 575fa8a3 (read-cache: read data in a hash-independent way, 2019-02-19) added a new code to copy from the on-disk data into the name member of the in-core cache entry, which is already done immediately after that in a way that takes prefix-compression into account. Remove this code, as it is not just unnecessary, but also can be reading beyond the on-disk data, when we are copying very long prefix string from the previous entry. Signed-off-by: ZheNing Hu <adlternative@gmail.com> [jc: rewrote the log message with Réne's findings] Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | Merge branch 'ds/sparse-sparse-checkout'Junio C Hamano2022-06-031-3/+3
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | "sparse-checkout" learns to work well with the sparse-index feature. * ds/sparse-sparse-checkout: sparse-checkout: integrate with sparse index p2000: add test for 'git sparse-checkout [add|set]' sparse-index: complete partial expansion sparse-index: partially expand directories sparse-checkout: --no-sparse-index needs a full index cache-tree: implement cache_tree_find_path() sparse-index: introduce partially-sparse indexes sparse-index: create expand_index() t1092: stress test 'git sparse-checkout set' t1092: refactor 'sparse-index contents' test
| * | | sparse-index: introduce partially-sparse indexesDerrick Stolee2022-05-231-3/+3
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A future change will present a temporary, in-memory mode where the index can both contain sparse directory entries but also not be completely collapsed to the smallest possible sparse directories. This will be necessary for modifying the sparse-checkout definition while using a sparse index. For now, convert the single-bit member 'sparse_index' in 'struct index_state' to be a an 'enum sparse_index_mode' with three modes: * INDEX_EXPANDED (0): No sparse directories exist. This is always the case for repositories that do not use cone-mode sparse-checkout. * INDEX_COLLAPSED: Sparse directories may exist. Files outside the sparse-checkout cone are reduced to sparse directory entries whenever possible. * INDEX_PARTIALLY_SPARSE: Sparse directories may exist. Some file entries outside the sparse-checkout cone may exist. Running convert_to_sparse() may further reduce those files to sparse directory entries. The main reason to store this extra information is to allow convert_to_sparse() to short-circuit when the index is already in INDEX_EXPANDED mode but to actually do the necessary work when in INDEX_PARTIALLY_SPARSE mode. The INDEX_PARTIALLY_SPARSE mode will be used in an upcoming change. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | read-cache: set sparsity when index is newVictoria Dye2022-05-101-1/+17
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the index read in 'do_read_index()' does not exist on-disk, mark the index "sparse" if the executing command does not require a full index and sparse index is otherwise enabled. Some commands (such as 'git stash -u') implicitly create a new index (when the 'GIT_INDEX_FILE' variable points to a non-existent file) and perform some operation on it. However, when this index is created, it isn't created with the same sparsity settings as the repo index. As a result, while these indexes may be sparse during the operation, they are always expanded before being written to disk. We can avoid that expansion by defaulting the index to "sparse", in which case it will only be expanded if the full index is needed. Note that the function 'set_new_index_sparsity()' is created despite having only a single caller because additional callers will be added in a subsequent patch. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'vd/mv-refresh-stat'Junio C Hamano2022-04-041-2/+15
|\ \ | |/ |/| | | | | | | | | | | "git mv" failed to refresh the cached stat information for the entry it moved. * vd/mv-refresh-stat: mv: refresh stat info for moved entry
| * mv: refresh stat info for moved entryVictoria Dye2022-03-291-2/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Update the stat info of the moved index entry in 'rename_index_entry_at()' if the entry is up-to-date with the index. Internally, 'git mv' uses 'rename_index_entry_at()' to move the source index entry to the destination. However, it directly copies the stat info of the original cache entry, which will not reflect the 'ctime' of the file renaming operation that happened as part of the move. If a file is otherwise up-to-date with the index, that difference in 'ctime' will make the entry appear out-of-date until the next index-refreshing operation (e.g., 'git status'). Some commands, such as 'git reset', use the cached stat information to determine whether a file is up-to-date; if this information is incorrect, the command will fail when it should pass. In order to ensure a moved entry is evaluated as 'up-to-date' when appropriate, refresh the destination index entry's stat info in 'git mv' if and only if the file is up-to-date. Note that the test added in 't7001-mv.sh' requires a "sleep 1" to ensure the 'ctime' of the file creation will be definitively older than the 'ctime' of the renamed file in 'git mv'. Reported-by: Maximilian Reichel <reichemn@icloud.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'ns/core-fsyncmethod'Junio C Hamano2022-03-251-6/+13
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace core.fsyncObjectFiles with two new configuration variables, core.fsync and core.fsyncMethod. * ns/core-fsyncmethod: core.fsync: documentation and user-friendly aggregate options core.fsync: new option to harden the index core.fsync: add configuration parsing core.fsync: introduce granular fsync control infrastructure core.fsyncmethod: add writeout-only mode wrapper: make inclusion of Windows csprng header tightly scoped