delta/git.git/Documentation/technical, branch jk/repack-pack-keep-objects

pack-bitmap: implement optional name_hash cache

2013-12-30T20:19:23+00:00

When we use pack bitmaps rather than walking the object
graph, we end up with the list of objects to include in the
packfile, but we do not know the path at which any tree or
blob objects would be found.

In a recently packed repository, this is fine. A fetch would
use the paths only as a heuristic in the delta compression
phase, and a fully packed repository should not need to do
much delta compression.

As time passes, though, we may acquire more objects on top
of our large bitmapped pack. If clients fetch frequently,
then they never even look at the bitmapped history, and all
works as usual. However, a client who has not fetched since
the last bitmap repack will have "have" tips in the
bitmapped history, but "want" newer objects.

The bitmaps themselves degrade gracefully in this
circumstance. We manually walk the more recent bits of
history, and then use bitmaps when we hit them.

But we would also like to perform delta compression between
the newer objects and the bitmapped objects (both to delta
against what we know the user already has, but also between
"new" and "old" objects that the user is fetching). The lack
of pathnames makes our delta heuristics much less effective.

This patch adds an optional cache of the 32-bit name_hash
values to the end of the bitmap file. If present, a reader
can use it to match bitmapped and non-bitmapped names during
delta compression.

Here are perf results for p5310:

Test origin/master HEAD^ HEAD
-------------------------------------------------------------------------------------------------
5310.2: repack to disk 36.81(37.82+1.43) 47.70(48.74+1.41) +29.6% 47.75(48.70+1.51) +29.7%
5310.3: simulated clone 30.78(29.70+2.14) 1.08(0.97+0.10) -96.5% 1.07(0.94+0.12) -96.5%
5310.4: simulated fetch 3.16(6.10+0.08) 3.54(10.65+0.06) +12.0% 1.70(3.07+0.06) -46.2%
5310.6: partial bitmap 36.76(43.19+1.81) 6.71(11.25+0.76) -81.7% 4.08(6.26+0.46) -88.9%

You can see that the time spent on an incremental fetch goes
down, as our delta heuristics are able to do their work.
And we save time on the partial bitmap clone for the same
reason.

Signed-off-by: Vicent Marti
Signed-off-by: Jeff King
Signed-off-by: Junio C Hamano

documentation: add documentation for the bitmap format

2013-12-30T20:19:22+00:00

This is the technical documentation for the JGit-compatible Bitmap v1
on-disk format.

Signed-off-by: Vicent Marti 
Signed-off-by: Jeff King 
Signed-off-by: Junio C Hamano

Merge branch 'mg/more-textconv'

2013-10-23T20:21:31+00:00

Make "git grep" and "git show" pay attention to --textconv when
dealing with blob objects.

* mg/more-textconv:
  grep: honor --textconv for the case rev:path
  grep: allow to use textconv filters
  t7008: demonstrate behavior of grep with textconv
  cat-file: do not die on --textconv without textconv filters
  show: honor --textconv for blobs
  diff_opt: track whether flags have been set explicitly
  t4030: demonstrate behavior of show with textconv

Merge branch 'ss/doclinks'

2013-09-17T18:42:54+00:00

When we converted many documents that were traditionally text-only
to be formatted to AsciiDoc, we did not update links that point at
them to refer to the formatted HTML files.

* ss/doclinks:
  Documentation: make AsciiDoc links always point to HTML files

Merge branch 'mn/doc-pack-heu-remove-dead-pastebin'

2013-09-12T21:41:47+00:00

* mn/doc-pack-heu-remove-dead-pastebin:
  remove dead pastebin link from pack-heuristics document

Merge branch 'jl/submodule-mv'

2013-09-09T21:36:15+00:00

"git mv A B" when moving a submodule A does "the right thing",
inclusing relocating its working tree and adjusting the paths in
the .gitmodules file.

* jl/submodule-mv: (53 commits)
  rm: delete .gitmodules entry of submodules removed from the work tree
  mv: update the path entry in .gitmodules for moved submodules
  submodule.c: add .gitmodules staging helper functions
  mv: move submodules using a gitfile
  mv: move submodules together with their work trees
  rm: do not set a variable twice without intermediate reading.
  t6131 - skip tests if on case-insensitive file system
  parse_pathspec: accept :(icase)path syntax
  pathspec: support :(glob) syntax
  pathspec: make --literal-pathspecs disable pathspec magic
  pathspec: support :(literal) syntax for noglob pathspec
  kill limit_pathspec_to_literal() as it's only used by parse_pathspec()
  parse_pathspec: preserve prefix length via PATHSPEC_PREFIX_ORIGIN
  parse_pathspec: make sure the prefix part is wildcard-free
  rename field "raw" to "_raw" in struct pathspec
  tree-diff: remove the use of pathspec's raw[] in follow-rename codepath
  remove match_pathspec() in favor of match_pathspec_depth()
  remove init_pathspec() in favor of parse_pathspec()
  remove diff_tree_{setup,release}_paths
  convert common_prefix() to use struct pathspec
  ...

Documentation: make AsciiDoc links always point to HTML files

2013-09-06T21:49:06+00:00

AsciiDoc's "link" is supposed to create hyperlinks for HTML output, so
prefer a "link" to point to an HTML file instead of a text file if an HTML
version of the file is being generated. For RelNotes, keep pointing to
text files as no equivalent HTML files are generated.

If appropriate, also update the link description to not contain the linked
file's extension.

Signed-off-by: Sebastian Schuberth 
Signed-off-by: Junio C Hamano

remove dead pastebin link from pack-heuristics document

2013-08-23T19:09:31+00:00

Signed-off-by: Michal Nazarewicz 
Acked-by: Jon Loeliger 
Signed-off-by: Junio C Hamano

Document the HTTP transport protocols

2013-08-21T18:37:53+00:00

Signed-off-by: Shawn O. Pearce 
Revised-by: Tay Ray Chuan 
Signed-off-by: Nguyễn Thái Ngọc Duy 
Signed-off-by: Junio C Hamano

Merge branch 'ob/typofixes'

2013-08-01T19:01:01+00:00

* ob/typofixes:
  many small typofixes