Add basic support to read pack-*.idx v1 and v2 files

The index data is mapped into memory and then scanned using a
binary search algorithm to locate the matching entry for the
supplied git_oid.  The standard fanout hash trick is applied to
reduce the search space by 8 iterations.

Since the v1 and v2 file formats differ in their search function,
due to the different layouts used for the object records, we use
two different search implementations and a virtual function pointer
to jump to the correct version of code for the current pack index.
The single function jump per-pack should be faster then computing
a branch point inside the inner loop of a common binary search.

To improve concurrency during read operations the pack lock is only
held while verifying the index is actually open, or while opening
the index for the first time.  This permits multiple concurrent
readers to scan through the same index.

If an invalid index file is opened we close it and mark the
git_pack's invalid bit to true.  The git_pack structure is kept
around in its parent git_packlist, but the invalid bit will cause
all future readers to skip over the pack entirely.  Pruning the
invalid entries is relatively unimportant because they shouldn't
be very common, a $GIT_DIRECTORY/objects/pack directory tends to
only have valid pack files.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

1 files changed, 19 insertions, 0 deletions

diff --git a/src/odb.h b/src/odb.h
new file mode 100644
index 000000000..2f205b20f
--- /dev/null
+++ b/src/odb.h
@@ -0,0 +1,19 @@
+#ifndef INCLUDE_odb_h__
+#define INCLUDE_odb_h__
+
+/** First 4 bytes of a pack-*.idx file header.
+ *
+ * Note this header exists only in idx v2 and later.  The idx v1
+ * file format does not have a magic sequence at the front, and
+ * must be detected by the first four bytes *not* being this value
+ * and the first 8 bytes matching the following expression:
+ *
+ *   uint32_t *fanout = ... the file data at offset 0 ...
+ *   ntohl(fanout[0]) < ntohl(fanout[1])
+ *
+ * The value chosen here for PACK_TOC is such that the above
+ * cannot be true for an idx v1 file.
+ */
+#define PACK_TOC 0xff744f63 /* -1tOc */
+
+#endif