summaryrefslogtreecommitdiff
path: root/src/indexer.c
Commit message (Collapse)AuthorAgeFilesLines
...
* Consolidate packfile allocation furtherRussell Belfer2013-04-221-1/+1
| | | | | | | Rename git_packfile_check to git_packfile_alloc since it is now being used more in that capacity. Fix the various places that use it. Consolidate some repeated code in odb_pack.c related to the allocation of a new pack_backend.
* Make indexer use shared packfile open codeRussell Belfer2013-04-221-24/+6
| | | | | | | | | | | The indexer was creating a packfile object separately from the code in pack.c which was a problem since I put a call to git_mutex_init into just pack.c. This commit updates the pack function for creating a new pack object (i.e. git_packfile_check()) so that it can be used in both places and then makes indexer.c use the shared initialization routine. There are also a few minor formatting and warning message fixes.
* Several warnings detected by static code analyzer fixedArkadiy Shapkin2013-03-181-3/+9
| | | | | | | Implicit type conversion argument of function to size_t type Suspicious sequence of types castings: size_t -> int -> size_t Consider reviewing the expression of the 'A = B == C' kind. The expression is calculated as following: 'A = (B == C)' Unsigned type is never < 0
* indexer: use a hashtable for keeping track of offsetsCarlos Martín Nieto2013-03-031-18/+25
| | | | | | | | | | These offsets are needed for REF_DELTA objects, which encode which object they use as a base, but not where it lies in the packfile, so we need a list. These objects are mostly from older packfiles, before OFS_DELTA was widely spread. The time spent in indexing these packfiles is greatly reduced, though remains above what git is able to do.
* indexer: kill git_indexerCarlos Martín Nieto2013-03-031-327/+1
| | | | | | | This was the first implementation and its goal was simply to have something that worked. It is slow and now it's just taking up space. Remove it and switch the one known usage to use the streaming indexer.
* Fix a git_filebuf leak (fixes Win32 clone::can_cancel)Philip Kelley2013-02-141-13/+14
|
* Allow all non-zero returns to cancel transfersBen Straub2013-02-051-1/+1
|
* Allow progress callback to cancel fetchBen Straub2013-02-051-6/+9
| | | | | This works by having the indexer watch the return code of the callback, so will only take effect on object boundaries.
* indexer: properly free the packfile resourcesCarlos Martín Nieto2013-01-121-4/+4
| | | | | | | | The indexer needs to call the packfile's free function so it takes care of freeing the caches. We still need to close the mwf descriptor manually so we can rename the packfile into its final name on Windows.
* Revert "pack: packfile_free -> git_packfile_free and use it in the indexers"Carlos Martín Nieto2013-01-111-2/+2
| | | | | | This reverts commit f289f886cb81bb570bed747053d5ebf8aba6bef7, which makes the tests fail on Windows. Revert until we can figure out a solution.
* Fix MSVC compilation warningsnulltoken2013-01-111-1/+1
|
* pack: packfile_free -> git_packfile_free and use it in the indexersCarlos Martín Nieto2013-01-111-2/+2
| | | | | It turns out the indexers have been ignoring the pack's free function and leaking data. Plug that.
* update copyrightsEdward Thomson2013-01-081-1/+1
|
* Fix MSVC compilation warningsnulltoken2012-12-171-3/+3
|
* indexer: move the temporary buffers into the indexer objectCarlos Martín Nieto2012-12-071-10/+9
| | | | | | Storing 4kB or 8kB in the stack is not very gentle. As this part has to be linear, put the buffer into the indexer object so we allocate it once in the heap.
* indexer: correctly deal with objects larger than the window sizeCarlos Martín Nieto2012-11-301-19/+26
| | | | | | | | A mmap-window is not guaranteed to give you the whole object, but the indexer currently assumes so. Loop asking for more data until we've successfully CRC'd all of the packed data.
* indexer: make use of streaming also for deltasCarlos Martín Nieto2012-11-301-49/+78
| | | | | | | | | | | | Up to now, deltas needed to be enterily in the packfile, and we tried to decompress then in their entirety over and over again. Adjust the logic so we read them as they come, just as we do for full objects. This also allows us to simplify the logic and have less nested code. The delta resolving phase still needs to decompress the whole object into memory, as there is not yet any streaming delta-apply support, but it helps in speeding up the downloading process and reduces the amount of memory allocations we need to do.
* indexer: use the packfile streaming APICarlos Martín Nieto2012-11-301-27/+130
| | | | | | | | | | | | | The new API allows us to read the object bit by bit from the packfile, instead of needing it all at once in the packfile. This also allows us to hash the object as it comes in from the network instead of having to try to read it all and failing repeatedly for larger objects. This is only the first step, but it already shows huge improvements when dealing with objects over a few megabytes in size. It reduces the memory needs in some cases, but delta objects still need to be completely in memory and the old inefficent method is still used for that.
* API updates for indexer.hBen Straub2012-11-271-2/+2
|
* Remove use of English expletivesMartin Woodward2012-11-231-1/+1
| | | | | | Remove words such as fuck, crap, shit etc. Remove other potentially offensive words from comments. Tidy up other geopolicital terms in comments.
* Indexer: Avoid a possible double-deletion in error caseSascha Cunz2012-11-181-2/+3
|
* Remove git_hash_ctx_new - callers now _ctx_init()Edward Thomson2012-11-131-12/+12
|
* Win32 CryptoAPI and CNG support for SHA1Edward Thomson2012-11-131-11/+17
|
* Fetch: don't clobber received countBen Straub2012-10-291-1/+2
| | | | | This memset was being reached after the entire packfile under WinHttp, so the byte count was being lost for small repos.
* Network progress: rename thingsBen Straub2012-10-241-18/+18
| | | | | | | git_indexer_stats and friends -> git_transfer_progress* Also made git_transfer_progress members more sanely named.
* Indexing progress now goes to 100%Ben Straub2012-10-191-0/+1
|
* Fetch/indexer: progress callbacksBen Straub2012-10-191-1/+18
|
* indexer: do not require absolute pathMichael Schubert2012-10-091-5/+0
|
* indexer: kill git_indexer_stats.data_receivedCarlos Martín Nieto2012-08-261-8/+0
| | | | | | | It's not really needed with the current code as we have EOS and the sideband's flush to tell us we're done. Keep the distinction between processed and received objects.
* indexer: don't segfault when freeing an unused indexerCarlos Martín Nieto2012-08-251-3/+5
| | | | | Make sure that idx->pack isn't NULL before trying to free resources under it.
* indexer: recognize and mark when all of the packfile has been downloadedCarlos Martín Nieto2012-08-241-1/+11
| | | | | We can't always rely on the network telling us when the download is finished. Recognize it from the indexer itself.
* indexer: delay resolving deltasCarlos Martín Nieto2012-07-141-22/+24
| | | | | | Not all delta bases are available on the first try. By delaying resolving all deltas until the end, we avoid decompressing some of the data twice or even more times, saving effort and time.
* mwindow: allow memory-window files to deregisterCarlos Martin Nieto2012-06-281-0/+2
| | | | | | | | | Once a file is registered, there is no way to deregister it, even after the structure that contains it is no longer needed and has been freed. This may be the source of #624. Allow and use the deregister function to remove our file from the global list.
* indexer: don't use '/objects/pack/' unconditionallyCarlos Martín Nieto2012-06-281-1/+1
| | | | | Not everyone who indexes a packfile wants to put it in the standard git repository location.
* indexer: start parsing input data immediatelyMichael Schubert2012-06-251-2/+0
| | | | | | | | | Currently, the first call of git_indexer_stream_add adds the data to the underlying pack file and opens it for later use, but doesn't start parsing the already available data. This means, git_indexer_stream_finalize only works if git_indexer_stream_add was called at least twice. Kill this limitation by parsing available data immediately.
* Merge remote-tracking branch 'source/development' into developmentChris Young2012-06-131-1/+3
|\
| * misc: Fix warnings from PVS Studio trialVicent Martí2012-06-071-1/+3
| |
* | Actually do the mmap... unsurprisingly, this makes the indexer work on SFSChris Young2012-06-121-1/+1
|/ | | | | On RAM: the .idx and .pack files become links to a .lock and the original download respectively. Assume some feature (such as record locking) supported by SFS but not JXFS or RAM: is required.
* CleanupMichael Schubert2012-05-191-1/+0
| | | | | | | * indexer: remove leftover printf * commit: remove unused macros COMMIT_BASIC_PARSE, COMMIT_FULL_PARSE and COMMIT_PRINT
* errors: Rename error codesbreaking-changesVicent Martí2012-05-181-4/+4
|
* errors: Rename the generic return codesVicent Martí2012-05-181-4/+4
|
* indexer: add more consistency checksCarlos Martín Nieto2012-05-161-0/+11
| | | | | Error out in finalize if there is junk after the packfile hash or we couldn't process all the objects.
* Introduce GITERR_INDEXERCarlos Martín Nieto2012-05-151-5/+5
|
* indexer: close the pack's fd before renaming itCarlos Martín Nieto2012-05-091-1/+2
| | | | Windows gets upset if we rename a file with an open descriptor.
* buf: deploy git_buf_len()nulltoken2012-04-301-2/+2
|
* Fix Win32 warningsRussell Belfer2012-04-261-1/+1
|
* transports: buffer the git requests before sending themCarlos Martín Nieto2012-04-251-1/+1
| | | | | | Trying to send every single line immediately won't give us any speed improvement and duplicates the code we need for other transports. Make the git transport use the same buffer functions as HTTP.
* indexer: add git_indexer_stream_free() and _hash()Carlos Martín Nieto2012-04-131-0/+29
|
* indexer: Add git_indexer_stream_finalize()Carlos Martín Nieto2012-04-131-71/+235
| | | | | Resolve any lingering deltas, write out the index file and rename the packfile.
* indexer: start writing the stream indexerCarlos Martín Nieto2012-04-131-36/+310
| | | | | | This will allow us to index a packfile as soon as we receive it from the network as well as storing it with its final name so we don't need to pass temporary file names around.
View Lorry Controller Status