summaryrefslogtreecommitdiff
path: root/src/odb_pack.c
Commit message (Collapse)AuthorAgeFilesLines
* midx: Support multi-pack-index files in odb_pack.clhchavez2020-11-271-106/+299
| | | | | | | | | | | | This change adds support for reading multi-pack-index files from the packfile odb backend. This also makes git_pack_file objects open their backing failes lazily in more scenarios, since the multi-pack-index can avoid having to open them in some cases (yay!). This change also refreshes the documentation found in src/odb_pack.c to match the updated code. Part of: #5399
* odb_pack: use GIT_ASSERTEdward Thomson2020-11-271-8/+18
|
* sha1_lookup: inline its only function into "pack.c"Patrick Steinhardt2020-02-071-1/+0
| | | | | | | The file "sha1_lookup.c" contains a single function `sha1_position` only which is used only in the packfile implementation. As the function is comparatively small, to enable the compiler to optimize better and to remove symbol visibility, move it into "pack.c".
* fileops: rename to "futils.h" to match function signaturesPatrick Steinhardt2019-07-201-1/+1
| | | | | | | | | Our file utils functions all have a "futils" prefix, e.g. `git_futils_touch`. One would thus naturally guess that their definitions and implementation would live in files "futils.h" and "futils.c", respectively, but in fact they live in "fileops.h". Rename the files to match expectations.
* indexer: use git_indexer_progress throughoutEdward Thomson2019-02-221-3/+3
| | | | | Update internal usage of `git_transfer_progress` to `git_indexer_progreses`.
* git_error: use new names in internal APIs and usageEdward Thomson2019-01-221-3/+3
| | | | | Move to the `git_error` name in the internal API for error-related functions.
* Fix odb foreach to also close on positive error codeMarijan Šuflaj2019-01-201-1/+1
| | | | | | | | In include/git2/odb.h it states that callback can also return positive value which should break looping. Implementations of git_odb_foreach() and pack_backend__foreach() did not respect that.
* object_type: use new enumeration namesethomson/index_fixesEdward Thomson2018-12-011-3/+3
| | | | Use the new object_type enumeration names within the codebase.
* Prevent heap-buffer-overflowbisho2018-09-051-1/+1
| | | | | | | | | | | | | | | | | When running repack while doing repo writes, `packfile_load__cb()` can see some temporary files in the directory that are bigger than the usual, and makes `memcmp` overflow on the `p->pack_name` string. ASAN detected this. This just uses `strncmp`, that should not have any performance impact and is safe for comparing strings of different sizes. ``` ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61200001a3f3 at pc 0x7f4a9e1976ec bp 0x7ffc1f80e100 sp 0x7ffc1f80d8b0 READ of size 89 at 0x61200001a3f3 thread T0 SCARINESS: 26 (multi-byte-read-heap-buffer-overflow) #0 0x7f4a9e1976eb in __interceptor_memcmp.part.78 (/build/cfgr-admin#link-tree/libtools_build_sanitizers_asan-ubsan-py.so+0xcf6eb) #1 0x7f4a518c5431 in packfile_load__cb /build/libgit2/0.27.0/src/libgit2-0.27.0/src/odb_pack.c:213 #2 0x7f4a518d9582 in git_path_direach /build/libgit2/0.27.0/src/libgit2-0.27.0/src/path.c:1134 #3 0x7f4a518c58ad in pack_backend__refresh /build/libgit2/0.27.0/src/libgit2-0.27.0/src/odb_pack.c:347 #4 0x7f4a518c1b12 in git_odb_refresh /build/libgit2/0.27.0/src/libgit2-0.27.0/src/odb.c:1511 #5 0x7f4a518bff5f in git_odb__freshen /build/libgit2/0.27.0/src/libgit2-0.27.0/src/odb.c:752 #6 0x7f4a518c17d4 in git_odb_stream_finalize_write /build/libgit2/0.27.0/src/libgit2-0.27.0/src/odb.c:1415 #7 0x7f4a51b9d015 in Repository_write /build/pygit2/0.27.0/src/pygit2-0.27.0/src/repository.c:509 ```
* odb_pack: fix passing partially initialized indexer optionsPatrick Steinhardt2018-06-221-1/+1
|
* indexer: introduce options struct to `git_indexer_new`Patrick Steinhardt2018-06-221-1/+5
| | | | | | | | | | We strive to keep an options structure to many functions to be able to extend options in the future without breaking the API. `git_indexer_new` doesn't have one right now, but we want to be able to add an option for enabling strict packfile verification. Add a new `git_indexer_options` structure and adjust callers to use that.
* Convert usage of `git_buf_free` to new `git_buf_dispose`Patrick Steinhardt2018-06-101-2/+2
|
* Make sure to always include "common.h" firstPatrick Steinhardt2017-07-031-0/+1
| | | | | | | | | | | | | | | | | | | | | | Next to including several files, our "common.h" header also declares various macros which are then used throughout the project. As such, we have to make sure to always include this file first in all implementation files. Otherwise, we might encounter problems or even silent behavioural differences due to macros or defines not being defined as they should be. So in fact, our header and implementation files should make sure to always include "common.h" first. This commit does so by establishing a common include pattern. Header files inside of "src" will now always include "common.h" as its first other file, separated by a newline from all the other includes to make it stand out as special. There are two cases for the implementation files. If they do have a matching header file, they will always include this one first, leading to "common.h" being transitively included as first file. If they do not have a matching header file, they instead include "common.h" as first file themselves. This fixes the outlined problems and will become our standard practice for header and source files inside of the "src/" from now on.
* odb_pack: initialize `git_rawobj` structurePatrick Steinhardt2017-03-211-1/+1
| | | | | | | | | | The `pack_entry_find_prefix` function receives a `git_rawobj` structure as argument. While the function first initializes the structure to a sensible state, Coverity is unable to correctly detect this, resulting in a warning. Fix this warning by initializing the object to all-zeroes before passing it to the function.
* odb: only freshen pack files every 2 secondsethomson/refresh_objectsEdward Thomson2016-08-041-1/+14
| | | | | | Since writing multiple objects may all already exist in a single packfile, avoid freshening that packfile repeatedly in a tight loop. Instead, only freshen pack files every 2 seconds.
* odb: freshen existing objects when writingEdward Thomson2016-08-041-0/+13
| | | | | | When writing an object, we calculate its OID and see if it exists in the object database. If it does, we need to freshen the file that contains it.
* mwindow: init mwindow files in git_libgit2_initethomson/mwindow_initEdward Thomson2016-08-041-3/+0
|
* delta: move delta application to delta.cEdward Thomson2016-05-261-1/+1
| | | | | | | Move the delta application functions into `delta.c`, next to the similar delta creation functions. Make the `git__delta_apply` functions adhere to other naming and parameter style within the library.
* odb: improved not found error messagesEdward Thomson2016-03-071-3/+5
| | | | | When looking up an abbreviated oid, show the actual (abbreviated) oid the caller passed instead of a full (but ambiguously truncated) oid.
* odb: Be smarter when refreshing backendsVicent Marti2015-10-141-79/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | In the current implementation of ODB backends, each backend is tasked with refreshing itself after a failed lookup. This is standard Git behavior: we want to e.g. reload the packfiles on disk in case they have changed and that's the reason we can't find the object we're looking for. This behavior, however, becomes pathological in repositories where multiple alternates have been loaded. Given that each alternate counts as a separate backend, a miss in the main repository (which can potentially be very frequent in cases where object storage comes from the alternate) will result in refreshing all its packfiles before we move on to the alternate backend where the object will most likely be found. To fix this, the code in `odb.c` has been refactored as to perform the refresh of all the backends externally, once we've verified that the object is nowhere to be found. If the refresh is successful, we then perform the lookup sequentially through all the backends, skipping the ones that we know for sure weren't refreshed (because they have no refresh API). The on-disk pack backend has been adjusted accordingly: it no longer performs refreshes internally.
* Initialize a few variablesCarlos Martín Nieto2015-06-091-1/+1
| | | | | | Coverity complains about the git_rawobj ones because we use a loop in which we keep remembering the old version, and we end up copying our object as the base, so we want to have the data pointer be NULL.
* Share packs across repository instancescmn/global-mwfCarlos Martín Nieto2014-06-231-3/+6
| | | | | | | | | | | Opening the same repository multiple times will currently open the same file multiple times, as well as map the same region of the file multiple times. This is not necessary, as the packfile data is immutable. Instead of opening and closing packfiles directly, introduce an indirection and allocate packfiles globally. This does mean locking on each packfile open, but we already use this lock for the global mwindow list so it doesn't introduce a new contention point.
* Don't redefine the same callback types, their signatures may changeJacques Germishuys2014-04-211-1/+1
|
* Add exists_prefix to ODB backend and ODB APIRussell Belfer2014-03-041-0/+18
|
* Remove converting user error to GIT_EUSERRussell Belfer2013-12-111-22/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes the behavior of callbacks so that the callback error code is not converted into GIT_EUSER and instead we propagate the return value through to the caller. Instead of using the giterr_capture and giterr_restore functions, we now rely on all functions to pass back the return value from a callback. To avoid having a return value with no error message, the user can call the public giterr_set_str or some such function to set an error message. There is a new helper 'giterr_set_callback' that functions can invoke after making a callback which ensures that some error message was set in case the callback did not set one. In places where the sign of the callback return value is meaningful (e.g. positive to skip, negative to abort), only the negative values are returned back to the caller, obviously, since the other values allow for continuing the loop. The hardest parts of this were in the checkout code where positive return values were overloaded as meaningful values for checkout. I fixed this by adding an output parameter to many of the internal checkout functions and removing the overload. This added some code, but it is probably a better implementation. There is some funkiness in the network code where user provided callbacks could be returning a positive or a negative value and we want to rely on that to cancel the loop. There are still a couple places where an user error might get turned into GIT_EUSER there, I think, though none exercised by the tests.
* Improve GIT_EUSER handlingRussell Belfer2013-12-111-21/+37
| | | | | | | | | | | This adds giterr_user_cancel to return GIT_EUSER and clear any error message that is sitting around. As a result of using that in places, we need to be more thorough with capturing errors that happen inside a callback when used internally. To help with that, this also adds giterr_capture and giterr_restore so that when we internally use a foreach-type function that clears errors and converts them to GIT_EUSER, it is easier to restore not just the return value, but the actual error message text.
* Allow callers to set mode on packfile creationEdward Thomson2013-11-071-1/+1
|
* indexer: remove the stream infixCarlos Martín Nieto2013-10-301-7/+7
| | | | | | | | | | | It was there to keep it apart from the one which read in from a file on disk. This other indexer does not exist anymore, so there is no need for anything other than git_indexer to refer to it. While here, rename _add() function to _append() and _finalize() to _commit(). The former change is cosmetic, while the latter avoids talking about "finalizing", which OO languages use to mean something completely different.
* Merge pull request #1891 from libgit2/cmn/fix-thin-packsVicent Martí2013-10-281-1/+2
|\ | | | | Add support for thin packs
| * indexer: fix thin packsCarlos Martín Nieto2013-10-041-1/+2
| | | | | | | | | | | | When given an ODB from which to read objects, the indexer will attempt to inject the missing bases at the end of the pack and update the header and trailer to reflect the new contents.
* | Initial iconv hookup for precomposed unicodeRussell Belfer2013-10-031-1/+1
|/ | | | | | | | | | | | | | | | | | | | This hooks up git_path_direach and git_path_dirload so that they will take a flag indicating if directory entry names should be tested and converted from decomposed unicode to precomposed form. This code will only come into play on the Apple platform and even then, only when certain types of filesystems are used. This involved adding a flag to these functions which involved changing a lot of places in the code. This was an opportunity to do a bit of code cleanup here and there, for example, getting rid of the git_futils_cleanupdir_r function in favor of a simple flag to git_futils_rmdir_r to not remove the top level entry. That ended up adding depth tracking during rmdir_r which led to a safety check for infinite directory recursion. Yay. This hasn't actually been tested on the Mac filesystems where the issue occurs. I still need to get test environment for that.
* path: Make direach() return EUSER on callback errornulltoken2013-09-101-1/+1
|
* odb: Move the auto refresh logic to the pack backendnulltoken2013-09-041-3/+73
| | | | | | | | | | | | Previously, `git_object_read()`, `git_object_read_prefix()` and `git_object_exists()` were implementing an auto refresh logic. When the expected object couldn't be found in any backend, a call to `git_odb_refresh()` was triggered and the lookup was once again performed against all backends. This commit removes this auto-refresh logic from the odb layer and pushes it down into the pack-backend (as it's the only one currently exposing a `refresh()` endpoint).
* odb: Code beautificationnulltoken2013-08-301-3/+6
|
* odb_pack: handle duplicate objects from different packsBrodie Rao2013-08-081-25/+16
| | | | | | | This is based on 24634c6fd02b2240e4a93fad70a08220f8fb793a. This also corrects an issue with error codes being mixed up with the number of found objects.
* Consolidate packfile allocation furtherRussell Belfer2013-04-221-42/+37
| | | | | | | Rename git_packfile_check to git_packfile_alloc since it is now being used more in that capacity. Fix the various places that use it. Consolidate some repeated code in odb_pack.c related to the allocation of a new pack_backend.
* Move odb_backend implementors stuff into git2/sysRussell Belfer2013-04-211-1/+2
| | | | | | | | | | | | This moves some of the odb_backend stuff that is related to the internals of an odb_backend implementation into include/git2/sys. Some of the stuff related to streaming I left in include/git2 because it seemed like it would be reasonably needed by a normal user who wanted to stream objects into and out of the ODB. Also, I added APIs for traversing the list of backends so that some of the tests would not need to access ODB internals.
* Several warnings detected by static code analyzer fixedArkadiy Shapkin2013-03-181-4/+4
| | | | | | | Implicit type conversion argument of function to size_t type Suspicious sequence of types castings: size_t -> int -> size_t Consider reviewing the expression of the 'A = B == C' kind. The expression is calculated as following: 'A = (B == C)' Unsigned type is never < 0
* odb_pack: Unused functionsVicent Marti2013-03-151-20/+0
|
* indexer: properly free the packfile resourcesCarlos Martín Nieto2013-01-121-1/+1
| | | | | | | | The indexer needs to call the packfile's free function so it takes care of freeing the caches. We still need to close the mwf descriptor manually so we can rename the packfile into its final name on Windows.
* Revert "pack: packfile_free -> git_packfile_free and use it in the indexers"Carlos Martín Nieto2013-01-111-1/+1
| | | | | | This reverts commit f289f886cb81bb570bed747053d5ebf8aba6bef7, which makes the tests fail on Windows. Revert until we can figure out a solution.
* pack: packfile_free -> git_packfile_free and use it in the indexersCarlos Martín Nieto2013-01-111-1/+1
| | | | | It turns out the indexers have been ignoring the pack's free function and leaking data. Plug that.
* dat errorcodeVicent Marti2013-01-101-0/+5
|
* Sane refresh logicVicent Marti2013-01-101-43/+32
| | | | | | | All the ODB backends have a specific refresh interface. When reading an object, first we attempt every single backend: if the read fails, then we refresh all the backends and retry the read one more time to see if the object has appeared.
* update copyrightsEdward Thomson2013-01-081-1/+1
|
* Merge pull request #1115 from ben/struct-versionsVicent Martí2012-12-051-0/+2
|\ | | | | Version info for public structs
| * Deploy versioned git_odb_backend structureBen Straub2012-11-301-0/+2
| |
* | odb-pack: resurrect pack_backend__read_headerDavid Michael Barr2012-12-031-10/+9
|/
* Make git_odb_foreach_cb take const paramRussell Belfer2012-11-271-1/+1
| | | | | This makes the first OID param of the ODB callback a const pointer and also propogates that change all the way to the backends.
* create callback to handle packs from fetch, move the indexer to odb_packEdward Thomson2012-11-051-0/+67
|
View Lorry Controller Status