delta/git.git, branch bc/unuse-packfile

Don't close pack fd when free'ing pack windows

2013-08-02T16:27:26+00:00

Now that close_one_pack() has been introduced to handle file
descriptor pressure, it is not strictly necessary to close the
pack file descriptor in unuse_one_window() when we're under memory
pressure.

Jeff King provided a justification for leaving the pack file open:

   If you close packfile descriptors, you can run into racy situations
   where somebody else is repacking and deleting packs, and they go away
   while you are trying to access them. If you keep a descriptor open,
   you're fine; they last to the end of the process. If you don't, then
   they disappear from under you.

   For normal object access, this isn't that big a deal; we just rescan
   the packs and retry. But if you are packing yourself (e.g., because
   you are a pack-objects started by upload-pack for a clone or fetch),
   it's much harder to recover (and we print some warnings).

Let's do so (or uh, not do so).

Signed-off-by: Brandon Casey 
Signed-off-by: Junio C Hamano

sha1_file: introduce close_one_pack() to close packs on fd pressure

2013-08-02T15:53:54+00:00

When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window.  If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed.  When this happens, git may exceed the number of file
descriptors permitted by the system.

This latter situation can occur in show-ref or receive-pack during ref
advertisement.  During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository.  If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated.  When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs.  Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:

   error: cannot create standard input pipe for rev-list: Too many open files
   error: Could not run 'git rev-list'

This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit.  If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object.  But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.

When we have file descriptor pressure, we just need to find an open
pack to close.  We can leave the existing mmap windows open.  If
additional windows need to be mapped into the pack file, it will be
reopened when necessary.  If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.

Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):

   * pack with oldest mtime and no allocated mmap windows
   * pack with the least-recently-used windows, i.e. the pack
     with the oldest most-recently-used window, where none of
     the windows are in use
   * pack with the least-recently-used windows

Signed-off-by: Brandon Casey 
Signed-off-by: Junio C Hamano

Merge branch 'maint-1.8.1' into maint-1.8.2

2013-07-03T22:26:53+00:00

* maint-1.8.1:
  git-config: update doc for --get with multiple values

git-config: update doc for --get with multiple values

2013-07-03T18:48:15+00:00

Since commit 00b347d (git-config: do not complain about duplicate
entries, 2012-10-23), "git config --get" does not exit with an error if
there are multiple values for the specified key but instead returns the
last value.  Update the documentation to reflect this.

Signed-off-by: John Keeping 
Signed-off-by: Junio C Hamano

t0070 "mktemp to unwritable directory" needs SANITY

2013-06-11T21:23:31+00:00

Use the SANITY prerequisite when testing if a temp file can
be created in a read only directory.
Skip the test under CYGWIN, or skip it under Unix/Linux when
it is run as root.

Signed-off-by: Torsten Bögershausen 
Signed-off-by: Junio C Hamano

pre-push.sample: Make the script executable

2013-06-11T18:22:00+00:00

githooks(5) says that "[...]the .sample files are executable by default"
which was not true.

Signed-off-by: Wieland Hoffmann 
Signed-off-by: Junio C Hamano

trivial: Add missing period in documentation

2013-05-28T22:15:29+00:00

Signed-off-by: Phil Hord 
Signed-off-by: Junio C Hamano

Git 1.8.2.3

2013-05-09T20:31:17+00:00

Signed-off-by: Junio C Hamano

Merge branch 'tr/copy-revisions-from-stdin' into maint

2013-05-09T19:42:17+00:00

* tr/copy-revisions-from-stdin:
  read_revisions_from_stdin: make copies for handle_revision_arg

t5004: avoid using tar for checking emptiness of archive

2013-05-09T19:41:31+00:00

Test 2 of t5004 checks if a supposedly empty tar archive really
contains no files.  24676f02 (t5004: fix issue with empty archive test
and bsdtar) removed our commit hash to make it work with bsdtar, but
the test still fails on NetBSD and OpenBSD, which use their own tar
that considers a tar file containing only NULs as broken.

Here's what the different archivers do when asked to create a tar
file without entries:

	$ uname -v
	NetBSD 6.0.1 (GENERIC)
	$ gtar --version | head -1
	tar (GNU tar) 1.26
	$ bsdtar --version
	bsdtar 2.8.4 - libarchive 2.8.4

	$ : >zero.tar
	$ perl -e 'print "\0" x 10240' >tenk.tar
	$ sha1 zero.tar tenk.tar
	SHA1 (zero.tar) = da39a3ee5e6b4b0d3255bfef95601890afd80709
	SHA1 (tenk.tar) = 34e163be8e43c5631d8b92e9c43ab0bf0fa62b9c

	$ : | tar cf - -T - | sha1
	da39a3ee5e6b4b0d3255bfef95601890afd80709
	$ : | gtar cf - -T - | sha1
	34e163be8e43c5631d8b92e9c43ab0bf0fa62b9c
	$ : | bsdtar cf - -T - | sha1
	34e163be8e43c5631d8b92e9c43ab0bf0fa62b9c

So NetBSD's native tar creates an empty file, while GNU tar and bsdtar
both give us 10KB of NULs -- just like git archive with an empty tree.
Now let's see how the archivers handle these two kinds of empty tar
files:

	$ tar tf zero.tar; echo $?
	tar: Unexpected EOF on archive file
	1
	$ gtar tf zero.tar; echo $?
	gtar: This does not look like a tar archive
	gtar: Exiting with failure status due to previous errors
	2
	$ bsdtar tf zero.tar; echo $?
	0

	$ tar tf tenk.tar; echo $?
	tar: Cannot identify format. Searching...
	tar: End of archive volume 1 reached
	tar: Sorry, unable to determine archive format.
	1
	$ gtar tf tenk.tar; echo $?
	0
	$ bsdtar tf tenk.tar; echo $?
	0

NetBSD's tar complains about both, bsdtar happily accepts any of them
and GNU tar doesn't like zero-length archive files.  So the safest
course of action is to stay with our block-of-NULs format which is
compatible with GNU tar and bsdtar, as we can't make NetBSD's native
tar happy anyway.

We can simplify our test, however, by taking tar out of the picture.
Instead of extracting the archive and checking for the non-presence of
files, check if the file has a size of 10KB and contains only NULs.
This makes t5004 pass on NetBSD and OpenBSD.

Signed-off-by: Rene Scharfe 
Signed-off-by: Junio C Hamano