<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/libgit2.git/src/diff_generate.c, branch ethomson/dockerv1</title>
<subtitle>github.com: libgit2/libgit2.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/'/>
<entry>
<title>diff_generate: validate oid file size</title>
<updated>2019-01-25T22:47:39+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2019-01-21T11:32:53+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=89bd4ddb33aa04a7f89223a7f0fd4f8123a3463d'/>
<id>89bd4ddb33aa04a7f89223a7f0fd4f8123a3463d</id>
<content type='text'>
Index entries are 32 bit unsigned ints, not `size_t`s.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Index entries are 32 bit unsigned ints, not `size_t`s.
</pre>
</div>
</content>
</entry>
<entry>
<title>blob: validate that blob sizes fit in a size_t</title>
<updated>2019-01-25T22:36:38+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2019-01-20T22:40:38+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=c6cac733c147ff800f78e7dff81f90d93369ea68'/>
<id>c6cac733c147ff800f78e7dff81f90d93369ea68</id>
<content type='text'>
Our blob size is a `git_off_t`, which is a signed 64 bit int.  This may
be erroneously negative or larger than `SIZE_MAX`.  Ensure that the blob
size fits into a `size_t` before casting.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Our blob size is a `git_off_t`, which is a signed 64 bit int.  This may
be erroneously negative or larger than `SIZE_MAX`.  Ensure that the blob
size fits into a `size_t` before casting.
</pre>
</div>
</content>
</entry>
<entry>
<title>git_error: use new names in internal APIs and usage</title>
<updated>2019-01-22T22:30:35+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2018-12-27T19:47:34+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=f673e232afe22eb865cdc915e55a2df6493f0fbb'/>
<id>f673e232afe22eb865cdc915e55a2df6493f0fbb</id>
<content type='text'>
Move to the `git_error` name in the internal API for error-related
functions.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Move to the `git_error` name in the internal API for error-related
functions.
</pre>
</div>
</content>
</entry>
<entry>
<title>object_type: use new enumeration names</title>
<updated>2018-12-01T11:54:57+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2018-11-28T14:26:57+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=168fe39bea3368972a8b1a33d5908e73bc790c18'/>
<id>168fe39bea3368972a8b1a33d5908e73bc790c18</id>
<content type='text'>
Use the new object_type enumeration names within the codebase.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Use the new object_type enumeration names within the codebase.
</pre>
</div>
</content>
</entry>
<entry>
<title>index: use new enum and structure names</title>
<updated>2018-12-01T10:46:44+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2018-11-28T13:31:06+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=18e71e6d597abe6c7feb666429c921bd19dc0ba8'/>
<id>18e71e6d597abe6c7feb666429c921bd19dc0ba8</id>
<content type='text'>
Use the new-style index names throughout our own codebase.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Use the new-style index names throughout our own codebase.
</pre>
</div>
</content>
</entry>
<entry>
<title>Merge pull request #4436 from pks-t/pks/packfile-stream-free</title>
<updated>2018-06-11T17:26:22+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2018-06-11T17:26:22+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=3be7301151e44537d2423386dc71f1d3e496c664'/>
<id>3be7301151e44537d2423386dc71f1d3e496c664</id>
<content type='text'>
pack: rename `git_packfile_stream_free`</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
pack: rename `git_packfile_stream_free`</pre>
</div>
</content>
</entry>
<entry>
<title>Convert usage of `git_buf_free` to new `git_buf_dispose`</title>
<updated>2018-06-10T17:34:37+00:00</updated>
<author>
<name>Patrick Steinhardt</name>
<email>ps@pks.im</email>
</author>
<published>2018-02-08T11:14:48+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=ecf4f33a4e327a91496f72816f9f02d923e5af05'/>
<id>ecf4f33a4e327a91496f72816f9f02d923e5af05</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix stash save bug with fast path index check</title>
<updated>2018-06-06T18:45:56+00:00</updated>
<author>
<name>David Turner</name>
<email>novalis@novalis.org</email>
</author>
<published>2018-06-04T16:56:08+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=5a7d454bf657fdf4d726f2319bf554ffd4486ed3'/>
<id>5a7d454bf657fdf4d726f2319bf554ffd4486ed3</id>
<content type='text'>
If the index contains stat data for a modified file, and the file is
not racily dirty, and there exists an untracked working tree directory
alphabetically after that file, and there are no other changes to the
repo, then git_stash_save would fail. It would confuse the untracked
working tree directory for the modified file, because they have the
same sha: zero.  The wt directory has a sha of zero because it's a
directory, and the file would have a zero sha because we wouldn't read
the file -- we would just know that it doesn't match the index.  To
fix this confusion, we simply check mode as well as SHA.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
If the index contains stat data for a modified file, and the file is
not racily dirty, and there exists an untracked working tree directory
alphabetically after that file, and there are no other changes to the
repo, then git_stash_save would fail. It would confuse the untracked
working tree directory for the modified file, because they have the
same sha: zero.  The wt directory has a sha of zero because it's a
directory, and the file would have a zero sha because we wouldn't read
the file -- we would just know that it doesn't match the index.  To
fix this confusion, we simply check mode as well as SHA.
</pre>
</div>
</content>
</entry>
<entry>
<title>diff_generate: avoid excessive stats of .gitattribute files</title>
<updated>2018-01-03T16:28:24+00:00</updated>
<author>
<name>Patrick Steinhardt</name>
<email>ps@pks.im</email>
</author>
<published>2018-01-03T16:07:36+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=d8896bda5c43616f3c755242703fce7c2a97ad67'/>
<id>d8896bda5c43616f3c755242703fce7c2a97ad67</id>
<content type='text'>
When generating a diff between two trees, for each file that is to be
diffed we have to determine whether it shall be treated as text or as
binary files. While git has heuristics to determine which kind of diff
to generate, users can also that default behaviour by setting or
unsetting the 'diff' attribute for specific files.

Because of that, we have to query gitattributes in order to determine
how to diff the current files. Instead of hitting the '.gitattributes'
file every time we need to query an attribute, which can get expensive
especially on networked file systems, we try to cache them instead. This
works perfectly fine for every '.gitattributes' file that is found, but
we hit cache invalidation problems when we determine that an attribuse
file is _not_ existing. We do create an entry in the cache for missing
'.gitattributes' files, but as soon as we hit that file again we
invalidate it and stat it again to see if it has now appeared.

In the case of diffing large trees with each other, this behaviour is
very suboptimal. For each pair of files that is to be diffed, we will
repeatedly query every directory component leading towards their
respective location for an attributes file. This leads to thousands or
even hundreds of thousands of wasted syscalls.

The attributes cache already has a mechanism to help in that scenario in
form of the `git_attr_session`. As long as the same attributes session
is still active, we will not try to re-query the gitmodules files at all
but simply retain our currently cached results. To fix our problem, we
can create a session at the top-most level, which is the initialization
of the `git_diff` structure, and use it in order to look up the correct
diff driver. As the `git_diff` structure is used to generate patches for
multiple files at once, this neatly solves our problem by retaining the
session until patches for all files have been generated.

The fix has been tested with linux.git by calling
`git_diff_tree_to_tree` and `git_diff_to_buf` with v4.10^{tree} and
v4.14^{tree}.

                | time    | .gitattributes stats
    without fix | 33.201s | 844614
    with fix    | 30.327s | 4441

While execution only improved by roughly 10%, the stat(3) syscalls for
.gitattributes files decreased by 99.5%. The benchmarks were quite
simple with best-of-three timings on Linux ext4 systems. One can assume
that for network based file systems the performance gain will be a lot
larger due to a much higher latency.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
When generating a diff between two trees, for each file that is to be
diffed we have to determine whether it shall be treated as text or as
binary files. While git has heuristics to determine which kind of diff
to generate, users can also that default behaviour by setting or
unsetting the 'diff' attribute for specific files.

Because of that, we have to query gitattributes in order to determine
how to diff the current files. Instead of hitting the '.gitattributes'
file every time we need to query an attribute, which can get expensive
especially on networked file systems, we try to cache them instead. This
works perfectly fine for every '.gitattributes' file that is found, but
we hit cache invalidation problems when we determine that an attribuse
file is _not_ existing. We do create an entry in the cache for missing
'.gitattributes' files, but as soon as we hit that file again we
invalidate it and stat it again to see if it has now appeared.

In the case of diffing large trees with each other, this behaviour is
very suboptimal. For each pair of files that is to be diffed, we will
repeatedly query every directory component leading towards their
respective location for an attributes file. This leads to thousands or
even hundreds of thousands of wasted syscalls.

The attributes cache already has a mechanism to help in that scenario in
form of the `git_attr_session`. As long as the same attributes session
is still active, we will not try to re-query the gitmodules files at all
but simply retain our currently cached results. To fix our problem, we
can create a session at the top-most level, which is the initialization
of the `git_diff` structure, and use it in order to look up the correct
diff driver. As the `git_diff` structure is used to generate patches for
multiple files at once, this neatly solves our problem by retaining the
session until patches for all files have been generated.

The fix has been tested with linux.git by calling
`git_diff_tree_to_tree` and `git_diff_to_buf` with v4.10^{tree} and
v4.14^{tree}.

                | time    | .gitattributes stats
    without fix | 33.201s | 844614
    with fix    | 30.327s | 4441

While execution only improved by roughly 10%, the stat(3) syscalls for
.gitattributes files decreased by 99.5%. The benchmarks were quite
simple with best-of-three timings on Linux ext4 systems. One can assume
that for network based file systems the performance gain will be a lot
larger due to a much higher latency.
</pre>
</div>
</content>
</entry>
<entry>
<title>diff_generate: fix unsetting diff flags</title>
<updated>2017-11-30T15:26:05+00:00</updated>
<author>
<name>Patrick Steinhardt</name>
<email>ps@pks.im</email>
</author>
<published>2017-11-30T15:12:48+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=5ca3f11592426f80c004cea5beefc0d05f23265b'/>
<id>5ca3f11592426f80c004cea5beefc0d05f23265b</id>
<content type='text'>
The macro `DIFF_FLAG_SET` can be used to set or unset a flag by
modifying the diff's bitmask. While the case of setting the flag is
handled correctly, the case of unsetting the flag was not. Instead of
inverting the flags, we are inverting the value which is used to decide
whether we want to set or unset the bits.

The value being used here is a simple `bool` which is `false`. As that
is being uplifted to `int` when getting the bitwise-complement, we will
end up retaining all bits inside of the bitmask. As that's only ever
used to set `GIT_DIFF_IGNORE_CASE`, we were actually always ignoring
case for generated diffs.

Fix that by instead getting the bitwise-complement of `FLAG`, not `VAL`.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The macro `DIFF_FLAG_SET` can be used to set or unset a flag by
modifying the diff's bitmask. While the case of setting the flag is
handled correctly, the case of unsetting the flag was not. Instead of
inverting the flags, we are inverting the value which is used to decide
whether we want to set or unset the bits.

The value being used here is a simple `bool` which is `false`. As that
is being uplifted to `int` when getting the bitwise-complement, we will
end up retaining all bits inside of the bitmask. As that's only ever
used to set `GIT_DIFF_IGNORE_CASE`, we were actually always ignoring
case for generated diffs.

Fix that by instead getting the bitwise-complement of `FLAG`, not `VAL`.
</pre>
</div>
</content>
</entry>
</feed>
