summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| * Implemented connect_with which includes all the slicing functions, which now ↵Sebastian Thiel2010-10-181-89/+118
| | | | | | | | operate on the delta stream data directly, its yet to be tested though, and I am afraid of this
| * Worked my way up to re-encoding delta chunks, connect_with method still ↵Sebastian Thiel2010-10-181-66/+243
| | | | | | | | needs some work, intermediate commit
| * Brutally made code compile, most of the major functions are still commented ↵Sebastian Thiel2010-10-171-182/+142
| | | | | | | | out, but it should just be a matter of time until its back and working
| * Intermediate commit, working my way through the code, step by step. Didn't ↵Sebastian Thiel2010-10-152-52/+144
| | | | | | | | even try to compile it yet
| * First adjustment to prepare the algorithm to work on deltastreams directly, ↵Sebastian Thiel2010-10-151-236/+93
|/ | | | without an intermediate conversion into far-too-large DeltaChunks. The new style just uses a single index, using one-fourth of the memory
* Updated draft with latest data, finished it by defining future ways to ↵Sebastian Thiel2010-10-151-8/+6
| | | | improve the algorithm
* Merge branch 'memcpy'Sebastian Thiel2010-10-141-60/+129
|\
| * removed some debug codeSebastian Thiel2010-10-141-5/+4
| |
| * Fixed tiny little bug that would cause our own chunk to be overridden before ↵Sebastian Thiel2010-10-141-5/+6
| | | | | | | | we make one last computation with its unaltered values
| * Implemented simple pre-pass to count offsets to help calculate where each ↵Sebastian Thiel2010-10-141-60/+129
|/ | | | chunk is going to be. This way, memmove becomes memcpy, and only one grow is required
* Added initial version of a document to show possible ways to stream delta ↵Sebastian Thiel2010-10-143-0/+105
| | | | data most efficiently
* Added new stream type which will request its size from its stream. This ↵Sebastian Thiel2010-10-132-6/+19
| | | | triggers full decompression, currently, but allows work to be delayed even further.If people want a stream, they usually read it anyway. Then it doesn't matter which attriubute they query first
* Merge branch 'performance'Sebastian Thiel2010-10-138-106/+1071
|\
| * Integrated new algorithm into the stream class, it will now be chosen ↵Sebastian Thiel2010-10-132-23/+33
| | | | | | | | depending on the context to figure out which one to use. For some reason, the c version that is slow for big files really rocks when its about small files. Its better than the respective c implementation of the normal delta apply
| * apply_delta now has a C implementation, which is only 25 percent faster for ↵Sebastian Thiel2010-10-133-9/+81
| | | | | | | | small files ( the overhead of all the rest is very high ), but 4 times faster for large files, where the enormous call overhead coming in with python really starts to show
| * Disabled new implementation in favor of the old one - all that's needed is a ↵Sebastian Thiel2010-10-132-2/+6
| | | | | | | | c implementation of apply delta data
| * Reverse Delta Application was a nice experiment, as it has one major flaw: ↵Sebastian Thiel2010-10-132-4/+8
| | | | | | | | Currently it integrates chunks from its base into the topmost delta chunk list, which causes plenty of mem-move operations. Plenty means, many many many, and its getting worse the more deltas you have of course. The algorithm was supposed to reduce the amount of memory activity, but failed at this point, making it worse than before. Probably it would just be fastest to implement the previous python algorithm, which swaps two buffers, in c
| * Now building a single module called _perf which contains all the performance ↵Sebastian Thiel2010-10-135-31/+12
| | | | | | | | enhancements, which increases loadtimes, less is more
| * Put delta-apply code into separate function. Would have preferred to to have ↵Sebastian Thiel2010-10-135-887/+911
| | | | | | | | just one dynamic module, lets see whether includes are possible
| * Fixed memory bug, it was a small tiny thing, as well as stupid.Sebastian Thiel2010-10-131-8/+7
| |
| * implemented memory compression, but got evil memory bug once again ... ↵Sebastian Thiel2010-10-131-8/+56
| | | | | | | | probably its just as stupid as previously
| * optimized reallocation count, which improves speed a little bit. Previously ↵Sebastian Thiel2010-10-131-22/+15
| | | | | | | | it would easily get into the habbit of reallocating the vector just to add a single item
| * Fixed integrity check function, finalized code, so far it is working, and ↵Sebastian Thiel2010-10-131-7/+65
| | | | | | | | its very efficient as well as the amount of data-copies is minimized
| * Fixed terrible bug, which happened due to a change of the size of the ↵Sebastian Thiel2010-10-131-5/+7
| | | | | | | | vector, but too early actually, so a memmove would use incorrect values
| * Currently there is a weird memory bug, valgrind says it is writing one byte ↵Sebastian Thiel2010-10-121-33/+82
| | | | | | | | too much. Perhaps its because of the use of PyMem
| * Improved performance of python implementation by 10 percent, just by ↵Sebastian Thiel2010-10-121-65/+17
| | | | | | | | removing function calls and object creations
| * Implemented everything about the merging of the bases into the topmost delta ↵Sebastian Thiel2010-10-121-9/+101
| | | | | | | | list. Its not yet working, but at least its not crashing
| * Enhanced memory handling within the delta-stream parsing method. Removed the ↵Sebastian Thiel2010-10-122-32/+60
| | | | | | | | base delta chunk vector, which was a reminder of old (python) times which are long gone
| * prepared the slicing, as well as a few accompanying methods. There is still ↵Sebastian Thiel2010-10-121-28/+166
| | | | | | | | quite a lot functionality missing
| * Now adding chunks to the vectors, next up is to implement the actual chunk ↵Sebastian Thiel2010-10-122-34/+71
| | | | | | | | merging
| * Apparently, the most serious memory bugs are fixed for now, lets get back to ↵Sebastian Thiel2010-10-122-17/+57
| | | | | | | | the actual thing
| * Wow, this was a lesson. My full hatred goes to python, and C, and everything ↵Sebastian Thiel2010-10-122-29/+34
| | | | | | | | ... cool if you control everything, but not cool if an Object_New call doesn't do anything for you - creating a new instance of an own type in python doesn't appear to be that easy after all, at least not if you want your initializers/new procs to be called
| * Weird bug causes crash, its memory related of course. GDB tells me where, ↵Sebastian Thiel2010-10-122-27/+58
| | | | | | | | but the why is still a mystery
| * DeltaChunkVector is now a separate structure. I wished I had c++, but ... ↵Sebastian Thiel2010-10-121-43/+83
| | | | | | | | its probably a good exercise
| * Implemented a few more functions, but I realize the vector implementation ↵Sebastian Thiel2010-10-121-13/+64
| | | | | | | | actually wants to be in a separate structure
| * Initial frame of the connect_delta method, which seems to do something. ↵Sebastian Thiel2010-10-114-3/+310
|/ | | | Debugging is hellish, you really have to use python exception to get information out of there, printf doesn't do anything for some reason
* Merge branch 'reversedeltaaggregation'Sebastian Thiel2010-10-113-108/+124
|\
| * Improved performance of delta_chunk_slice method a tiny bit, but it really ↵Sebastian Thiel2010-10-111-2/+2
| | | | | | | | needs to go to c
| * Removed previous non-reverse delta-application functionality. Although it ↵Sebastian Thiel2010-10-112-111/+32
| | | | | | | | was slightly faster, this new version only needs a faster slicing, which consumes ridiculous amounts of time
| * Removed debugging codeSebastian Thiel2010-10-111-19/+2
| |
| * Reverse delta aggregration appears to be workingSebastian Thiel2010-10-112-50/+107
| |
| * New Frame uses a distinct type to express the different mode of operation. ↵Sebastian Thiel2010-10-111-23/+62
| | | | | | | | This is clean enough to get going
| * First frame to implement the actual data aggregation, but ... its probbaly ↵Sebastian Thiel2010-10-112-16/+32
|/ | | | going to change quite a lot again
* Merge branch 'deltaaggregation'Sebastian Thiel2010-10-117-14/+499
|\
| * Made heavliy called methods global, its brings a second, which is nearly 10 ↵Sebastian Thiel2010-10-111-75/+79
| | | | | | | | percent more performance just by eliminating two method calls
| * First profiling run revealed that the copy function was a serious slowdown. ↵Sebastian Thiel2010-10-112-7/+9
| | | | | | | | Now its twice as fast compared to the previous version, but still about 8 times slower than the brute force approach
| * Disabled delta-aggregation as it is reduces the throughput to 540KiB/s ↵Sebastian Thiel2010-10-102-7/+8
| | | | | | | | | | | | compared to 9.4MiB compared to the previous brute-force algorithm. Compression helps, but it would probably be more efficient if done right away, not as post-process. It might help to implement the reversed version of this algorithm, as initially intended, but currently the overhead is the actual application
| * implemented binary tree search to get the closest deltachunk by offsetSebastian Thiel2010-10-102-7/+13
| |
| * Implemented add-chunk compression, which clearly reduces chain size, but ↵Sebastian Thiel2010-10-102-38/+61
| | | | | | | | might not really be worth it in python
| * Forward Delta Application now appears to workSebastian Thiel2010-10-102-291/+146
| |
View Lorry Controller Status