delta/git.git, branch ls/filter-process

t0021: fix flaky test

2016-12-18T21:01:20+00:00

t0021.15 creates files, adds them to the index, and commits them. All
this usually happens in a test run within the same second and Git cannot
know if the files have been changed between `add` and `commit`.  Thus,
Git has to run the clean filter in both operations. Sometimes these
invocations spread over two different seconds and Git can infer that the
files were not changed between `add` and `commit` based on their
modification timestamp. The test would fail as it expects the filter
invocation. Remove this expectation to make the test stable.

Signed-off-by: Lars Schneider 
Signed-off-by: Junio C Hamano

docs: warn about possible '=' in clean/smudge filter process values

2016-12-06T19:29:52+00:00

A pathname value in a clean/smudge filter process "key=value" pair can
contain the '=' character (introduced in edcc858). Make the user aware
of this issue in the docs, add a corresponding test case, and fix the
issue in filter process value parser of the example implementation in
contrib.

Signed-off-by: Lars Schneider 
Signed-off-by: Junio C Hamano

t0021: remove debugging cruft

2016-11-11T21:09:24+00:00

The redirection of the standard error stream to a temporary file is
a leftover cruft during debugging.  Remove it.

Besides, it is reported by folks on the Windows that the test is
flaky with this redirection; somebody gets confused and this
merely-redirected-to file gets marked as delete-pending by git.exe
and makes it finish with a non-zero exit status when "git checkout"
finishes.  Windows folks may want to figure that one out, but for
the purpose of this test, it shouldn't become a show-stopper.

Signed-off-by: Junio C Hamano

t0021: compute file size with a single process instead of a pipeline

2016-11-08T20:26:40+00:00

Avoid unwanted coding patterns (prodigal use of pipelines), and in
particular a useless use of cat.

Signed-off-by: Johannes Sixt 
Signed-off-by: Jeff King

t0021: expect more variations in the output of uniq -c

2016-11-08T20:26:40+00:00

Some versions of uniq -c write the count left-justified, other version
write it right-justified. Be prepared for both kinds.

Signed-off-by: Johannes Sixt 
Signed-off-by: Jeff King

contrib/long-running-filter: add long running filter example

2016-10-17T18:45:52+00:00

Add a simple pass-thru filter as example implementation for the Git
filter protocol version 2. See Documentation/gitattributes.txt, section
"Filter Protocol" for more info.

Signed-off-by: Lars Schneider 
Signed-off-by: Junio C Hamano

convert: add filter..process option

2016-10-17T18:45:52+00:00

Git's clean/smudge mechanism invokes an external filter process for
every single blob that is affected by a filter. If Git filters a lot of
blobs then the startup time of the external filter processes can become
a significant part of the overall Git execution time.

In a preliminary performance test this developer used a clean/smudge
filter written in golang to filter 12,000 files. This process took 364s
with the existing filter mechanism and 5s with the new mechanism. See
details here: https://github.com/github/git-lfs/pull/1382

This patch adds the `filter..process` string option which, if
used, keeps the external filter process running and processes all blobs
with the packet format (pkt-line) based protocol over standard input and
standard output. The full protocol is explained in detail in
`Documentation/gitattributes.txt`.

A few key decisions:

* The long running filter process is referred to as filter protocol
  version 2 because the existing single shot filter invocation is
  considered version 1.
* Git sends a welcome message and expects a response right after the
  external filter process has started. This ensures that Git will not
  hang if a version 1 filter is incorrectly used with the
  filter..process option for version 2 filters. In addition,
  Git can detect this kind of error and warn the user.
* The status of a filter operation (e.g. "success" or "error) is set
  before the actual response and (if necessary!) re-set after the
  response. The advantage of this two step status response is that if
  the filter detects an error early, then the filter can communicate
  this and Git does not even need to create structures to read the
  response.
* All status responses are pkt-line lists terminated with a flush
  packet. This allows us to send other status fields with the same
  protocol in the future.

Helped-by: Martin-Louis Bright 
Reviewed-by: Jakub Narebski 
Signed-off-by: Lars Schneider 
Signed-off-by: Junio C Hamano

convert: prepare filter..process option

2016-10-17T18:36:50+00:00

Refactor the existing 'single shot filter mechanism' and prepare the
new 'long running filter mechanism'.

Signed-off-by: Lars Schneider 
Signed-off-by: Junio C Hamano

convert: make apply_filter() adhere to standard Git error handling

2016-10-17T18:36:50+00:00

apply_filter() returns a boolean that tells the caller if it
"did convert or did not convert". The variable `ret` was used throughout
the function to track errors whereas `1` denoted success and `0`
failure. This is unusual for the Git source where `0` denotes success.

Rename the variable and flip its value to make the function easier
readable for Git developers.

Signed-off-by: Lars Schneider 
Signed-off-by: Junio C Hamano

pkt-line: add functions to read/write flush terminated packet streams

2016-10-17T18:36:50+00:00

write_packetized_from_fd() and write_packetized_from_buf() write a
stream of packets. All content packets use the maximal packet size
except for the last one. After the last content packet a `flush` control
packet is written.

read_packetized_to_strbuf() reads arbitrary sized packets until it
detects a `flush` packet.

Signed-off-by: Lars Schneider 
Signed-off-by: Junio C Hamano