Running tests with '-x' option to trace the individual command
executions is a useful way to debug test scripts, but some tests
that capture the standard error stream and check what the command
said can be broken with the trace output mixed in.  When running
our tests under "bash", however, we can redirect the trace output
to another file descriptor to keep the standard error of programs
being tested intact.

* jk/test-send-sh-x-trace-elsewhere:
  test-lib: set BASH_XTRACEFD automatically

Passing "-x" to a test script enables the shell's "set -x"
tracing, which can help with tracking down the command that
is causing a failure. Unfortunately, it can also _cause_
failures in some tests that redirect the stderr of a shell
function.  Inside the function the shell continues to
respect "set -x", and the trace output is collected along
with whatever stderr is generated normally by the function.

You can see an example of this by running:

  ./t0040-parse-options.sh -x -i

which will fail immediately in the first test, as it
expects:

  test_must_fail some-cmd 2>output.err

to leave output.err empty (but with "-x" it has our trace
output).

Unfortunately there isn't a portable or scalable solution to
this. We could teach test_must_fail to disable "set -x", but
that doesn't help any of the other functions or subshells.

However, we can work around it by pointing the "set -x"
output to our descriptor 4, which always points to the
original stderr of the test script. Unfortunately this only
works for bash, but it's better than nothing (and other
shells will just ignore the BASH_XTRACEFD variable).

The patch itself is a simple one-liner, but note the caveats
in the accompanying comments.

Automatic tests for our "-x" option may be a bit too meta
(and a pain, because they are bash-specific), but I did
confirm that it works correctly both with regular "-x" and
with "--verbose-only=1". This works because the latter flips
"set -x" off and on for particular tests (if it didn't, we
would get tracing for all tests, as going to descriptor 4
effectively circumvents the verbose flag).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

When prefixing a Git call in the test suite with 'debug ', it will
now be run with GDB, allowing the developer to debug test failures
more conveniently.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

It's easy to miss an "&&"-chain in a test script, like:

  test_expect_success 'check something important' '
	cmd1 &&
	cmd2
	cmd3
  '

The test harness will notice if cmd3 fails, but a failure of
cmd1 or cmd2 will go unnoticed, as their exit status is lost
after cmd3 runs.

The toy example above is easy to spot because the "cmds" are
all the same length, but real code is much more complicated.
It's also difficult to detect these situations by statically
analyzing the shell code with regexps (like the
check-non-portable-shell script does); there's too much
context required to know whether a &&-chain is appropriate
on a given line or not.

This patch instead lets the shell check each test by
sticking a command with a specific and unusual return code
at the top of each test, like:

  (exit 117) &&
  cmd1 &&
  cmd2
  cmd3

In a well-formed test, the non-zero exit from the first
command prevents any of the rest from being run, and the
test's exit code is 117. In a bad test (like the one above),
the 117 is lost, and cmd3 is run.

When we encounter a failure of this check, we abort the test
script entirely. For one thing, we have no clue which subset
of the commands in the test snippet were actually run.
Running further tests would be pointless, because we're now
in an unknown state. And two, this is not a "test failure"
in the traditional sense. The test script is buggy, not the
code it is testing. We should be able to fix these problems
in the script once, and not have them come back later as a
regression in git's code.

After checking a test snippet for --chain-lint, we do still
run the test itself.  We could actually have a pure-lint
mode which just checks each test, but there are a few
reasons not to. One, because the tests are executing
arbitrary code, which could impact the later environment
(e.g., that could impact which set of tests we run at all).
And two, because a pure-lint mode would still be expensive
to run, because a significant amount of code runs outside of
the test_expect_* blocks.  Instead, this option is designed
to be used as part of a normal test suite run, where it adds
very little overhead.

Turning on this option detects quite a few problems in
existing tests, which will be fixed in subsequent patches.
However, there are a number of places it cannot reach:

 - it cannot find a failure to break out of loops on error,
   like:

     cmd1 &&
     for i in a b c; do
	     cmd2 $i
     done &&
     cmd3

   which will not notice failures of "cmd2 a" or "cmd b"

 - it cannot find a missing &&-chain inside a block or
   subfunction, like:

     foo () {
	     cmd1
	     cmd2
     }

     foo &&
     bar

   which will not notice a failure of cmd1.

 - it only checks tests that you run; every platform will
   have some tests skipped due to missing prequisites,
   so it's impossible to say from one run that the test
   suite is free of broken &&-chains. However, all tests get
   run by _somebody_, so eventually we will notice problems.

 - it does not operate on test_when_finished or prerequisite
   blocks. It could, but these tends to be much shorter and
   less of a problem, so I punted on them in this patch.

This patch was inspired by an earlier patch by Jonathan
Nieder:

  http://article.gmane.org/gmane.comp.version-control.git/235913

This implementation and all bugs are mine.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

* jc/refer-to-t-readme-from-submitting-patches:
  t/README: justify why "! grep foo" is sufficient
  SubmittingPatches: refer to t/README for tests

We require use of test_must_fail to check expected non-zero exit by
Git itself, but discourage test_must_fail to be used for checking
exit status of non Git commands that are supplied by the system.
The current text explains the reason for the former but not the
latter.

Signed-off-by: Junio C Hamano <gitster@pobox.com>

Usually running a test under "-v" makes it clear which
command is failing. However, sometimes it can be useful to
also see a complete trace of the shell commands being run in
the test. You can do so without any support from the test
suite by running "sh -x tXXXX-foo.sh". However, this
produces quite a large bit of output, as we see a trace of
the entire test suite.

This patch instead introduces a "-x" option to the test
scripts (i.e., "./tXXXX-foo.sh -x"). When enabled, this
turns on "set -x" only for the tests themselves. This can
still be a bit verbose, but should keep things to a more
manageable level. You can even use "--verbose-only" to see
the trace only for a specific test.

The implementation is a little invasive. We turn on the "set
-x" inside the "eval" of the test code. This lets the eval
itself avoid being reported in the trace (which would be
long, and redundant with the verbose listing we already
showed). And then after the eval runs, we do some trickery
with stderr to avoid showing the "set +x" to the user.

We also show traces for test_cleanup functions (since they
can impact the test outcome, too). However, we do avoid
running the noop ":" cleanup (the default if the test does
not use test_cleanup at all), as it creates unnecessary
noise in the "set -x" output.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Allow specifying only certain individual test pieces to be run
using a range notation (e.g. "t1234-test.sh --run='1-4 6 8 9-'").

* ib/test-selectively-run:
  t0000-*.sh: fix the GIT_SKIP_TESTS sub-tests
  test-lib: '--run' to run only specific tests
  test-lib: tests skipped by GIT_SKIP_TESTS say so
  test-lib: document short options in t/README

test_write_lines carefully quotes its arguments as "$@", so

	test_write_lines "a b" c

writes two lines as requested, not three.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

API and implementation as suggested by Junio.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>