<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/libgit2.git/tests/pack/filelimit.c, branch ethomson/codespaces</title>
<subtitle>github.com: libgit2/libgit2.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/'/>
<entry>
<title>str: introduce `git_str` for internal, `git_buf` is external</title>
<updated>2021-10-17T13:49:01+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2021-09-07T21:53:49+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=f0e693b18afbe1de37d7da5b5a8967b6c87d8e53'/>
<id>f0e693b18afbe1de37d7da5b5a8967b6c87d8e53</id>
<content type='text'>
libgit2 has two distinct requirements that were previously solved by
`git_buf`.  We require:

1. A general purpose string class that provides a number of utility APIs
   for manipulating data (eg, concatenating, truncating, etc).
2. A structure that we can use to return strings to callers that they
   can take ownership of.

By using a single class (`git_buf`) for both of these purposes, we have
confused the API to the point that refactorings are difficult and
reasoning about correctness is also difficult.

Move the utility class `git_buf` to be called `git_str`: this represents
its general purpose, as an internal string buffer class.  The name also
is an homage to Junio Hamano ("gitstr").

The public API remains `git_buf`, and has a much smaller footprint.  It
is generally only used as an "out" param with strict requirements that
follow the documentation.  (Exceptions exist for some legacy APIs to
avoid breaking callers unnecessarily.)

Utility functions exist to convert a user-specified `git_buf` to a
`git_str` so that we can call internal functions, then converting it
back again.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
libgit2 has two distinct requirements that were previously solved by
`git_buf`.  We require:

1. A general purpose string class that provides a number of utility APIs
   for manipulating data (eg, concatenating, truncating, etc).
2. A structure that we can use to return strings to callers that they
   can take ownership of.

By using a single class (`git_buf`) for both of these purposes, we have
confused the API to the point that refactorings are difficult and
reasoning about correctness is also difficult.

Move the utility class `git_buf` to be called `git_str`: this represents
its general purpose, as an internal string buffer class.  The name also
is an homage to Junio Hamano ("gitstr").

The public API remains `git_buf`, and has a much smaller footprint.  It
is generally only used as an "out" param with strict requirements that
follow the documentation.  (Exceptions exist for some legacy APIs to
avoid breaking callers unnecessarily.)

Utility functions exist to convert a user-specified `git_buf` to a
`git_str` so that we can call internal functions, then converting it
back again.
</pre>
</div>
</content>
</entry>
<entry>
<title>Make the pack and mwindow implementations data-race-free</title>
<updated>2020-11-29T03:40:56+00:00</updated>
<author>
<name>lhchavez</name>
<email>lhchavez@lhchavez.com</email>
</author>
<published>2020-08-02T01:24:41+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=322c15ee858622f2e3def514d3e7e1b47023950e'/>
<id>322c15ee858622f2e3def514d3e7e1b47023950e</id>
<content type='text'>
This change fixes a packfile heap corruption that can happen when
interacting with multiple packfiles concurrently across multiple
threads. This is exacerbated by setting a lower mwindow open file limit.

This change:

* Renames most of the internal methods in pack.c to clearly indicate
  that they expect to be called with a certain lock held, making
  reasoning about the state of locks a bit easier.
* Splits the `git_pack_file` lock in two: the one in `git_pack_file`
  only protects the `index_map`. The protection to `git_mwindow_file` is
  now in that struct.
* Explicitly checks for freshness of the `git_pack_file` in
  `git_packfile_unpack_header`: this allows the mwindow implementation
  to close files whenever there is enough cache pressure, and
  `git_packfile_unpack_header` will reopen the packfile if needed.
* After a call to `p_munmap()`, the `data` and `len` fields are poisoned
  with `NULL` to make use-after-frees more evident and crash rather than
  being open to the possibility of heap corruption.
* Adds a test case to prevent this from regressing in the future.

Fixes: #5591
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This change fixes a packfile heap corruption that can happen when
interacting with multiple packfiles concurrently across multiple
threads. This is exacerbated by setting a lower mwindow open file limit.

This change:

* Renames most of the internal methods in pack.c to clearly indicate
  that they expect to be called with a certain lock held, making
  reasoning about the state of locks a bit easier.
* Splits the `git_pack_file` lock in two: the one in `git_pack_file`
  only protects the `index_map`. The protection to `git_mwindow_file` is
  now in that struct.
* Explicitly checks for freshness of the `git_pack_file` in
  `git_packfile_unpack_header`: this allows the mwindow implementation
  to close files whenever there is enough cache pressure, and
  `git_packfile_unpack_header` will reopen the packfile if needed.
* After a call to `p_munmap()`, the `data` and `len` fields are poisoned
  with `NULL` to make use-after-frees more evident and crash rather than
  being open to the possibility of heap corruption.
* Adds a test case to prevent this from regressing in the future.

Fixes: #5591
</pre>
</div>
</content>
</entry>
<entry>
<title>runtime: move init/shutdown into the "runtime"</title>
<updated>2020-10-11T19:13:04+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2020-05-15T10:47:09+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=e316b0d3d64eb8f65f4109c1565d929b29e1d33a'/>
<id>e316b0d3d64eb8f65f4109c1565d929b29e1d33a</id>
<content type='text'>
Provide a mechanism for system components to register for initialization
and shutdown of the libgit2 runtime.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Provide a mechanism for system components to register for initialization
and shutdown of the libgit2 runtime.
</pre>
</div>
</content>
</entry>
<entry>
<title>Minor nits and style formatting</title>
<updated>2020-07-12T16:53:10+00:00</updated>
<author>
<name>lhchavez</name>
<email>lhchavez@lhchavez.com</email>
</author>
<published>2020-07-12T16:53:10+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=92d42eb3d83a28febbbb50df7c398e32677da28a'/>
<id>92d42eb3d83a28febbbb50df7c398e32677da28a</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Create the repository within the test</title>
<updated>2020-06-27T14:38:02+00:00</updated>
<author>
<name>lhchavez</name>
<email>lhchavez@lhchavez.com</email>
</author>
<published>2020-06-27T14:34:36+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=d88994da6058f4dc8260e3c16898ad7ab1b61855'/>
<id>d88994da6058f4dc8260e3c16898ad7ab1b61855</id>
<content type='text'>
This change moves the humongous static repository with 1025 commits into
the test file, now with a more modest 16 commits.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This change moves the humongous static repository with 1025 commits into
the test file, now with a more modest 16 commits.
</pre>
</div>
</content>
</entry>
<entry>
<title>Review feedback</title>
<updated>2020-06-26T23:45:25+00:00</updated>
<author>
<name>lhchavez</name>
<email>lhchavez@lhchavez.com</email>
</author>
<published>2020-06-26T23:10:30+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=eab2b0448e47801090c50fe9bb9e72e377f0317a'/>
<id>eab2b0448e47801090c50fe9bb9e72e377f0317a</id>
<content type='text'>
* Change the default of the file limit to 0 (unlimited).
* Changed the heuristic to close files to be the file that contains the
  least-recently-used window such that the window is the
  most-recently-used in the file, and the file does not have in-use
  windows.
* Parameterized the filelimit test to check for a limit of 1 and 100
  open windows.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* Change the default of the file limit to 0 (unlimited).
* Changed the heuristic to close files to be the file that contains the
  least-recently-used window such that the window is the
  most-recently-used in the file, and the file does not have in-use
  windows.
* Parameterized the filelimit test to check for a limit of 1 and 100
  open windows.
</pre>
</div>
</content>
</entry>
<entry>
<title>mwindow: set limit on number of open files</title>
<updated>2020-06-21T14:15:26+00:00</updated>
<author>
<name>lhchavez</name>
<email>lhchavez@lhchavez.com</email>
</author>
<published>2020-02-08T20:47:24+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/libgit2.git/commit/?id=9679df573604571d1372d2850116e66e1f4cec23'/>
<id>9679df573604571d1372d2850116e66e1f4cec23</id>
<content type='text'>
There are some cases in which repositories accrue a large number of
packfiles. The existing mwindow limit applies only to the total size of
mmap'd files, not on their number. This leads to a situation in which
having lots of small packfiles could exhaust the allowed number of open
files, particularly on macOS, where the default ulimit is very low
(256).

This change adds a new configuration parameter
(GIT_OPT_SET_MWINDOW_FILE_LIMIT) that sets the maximum number of open
packfiles, with a default of 128. This is low enough so that even macOS
users should not hit it during normal use.

Based on PR #5386, originally written by @josharian.

Fixes: #2758
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
There are some cases in which repositories accrue a large number of
packfiles. The existing mwindow limit applies only to the total size of
mmap'd files, not on their number. This leads to a situation in which
having lots of small packfiles could exhaust the allowed number of open
files, particularly on macOS, where the default ulimit is very low
(256).

This change adds a new configuration parameter
(GIT_OPT_SET_MWINDOW_FILE_LIMIT) that sets the maximum number of open
packfiles, with a default of 128. This is low enough so that even macOS
users should not hit it during normal use.

Based on PR #5386, originally written by @josharian.

Fixes: #2758
</pre>
</div>
</content>
</entry>
</feed>
