delta/go-git.git/src/archive/tar, branch dev.inline

archive/tar: disable prefix field in Writer

2016-11-02T20:18:38+00:00

The proper fix for the Writer is too involved to be done in time
for Go 1.8. Instead, we do a localized fix that simply disables the
prefix encoding logic. While this will prevent some legitimate uses
of prefix, it will ensure that we don't keep outputting invalid
GNU format files that have the prefix field populated.

For headers with long filenames that could have used the prefix field,
they will be promoted to use the PAX format, which ensures that we
will still be able to encode all headers that we were able to do before.

Updates #12594
Fixes #17630
Fixes #9683

Change-Id: Ia97b524ac69865390e2ae8bb0dfb664d40a05add
Reviewed-on: https://go-review.googlesource.com/32234
Reviewed-by: Russ Cox 
Run-TryBot: Joe Tsai 
TryBot-Result: Gobot Gobot

archive/tar: validate sparse headers in parsePAX

2016-10-22T16:35:14+00:00

According to the GNU manual, the format is:
<<<
GNU.sparse.size=size
GNU.sparse.numblocks=numblocks
repeat numblocks times
  GNU.sparse.offset=offset
  GNU.sparse.numbytes=numbytes
end repeat
>>>

The logic in parsePAX converts the repeating sequence of
(offset, numbytes) pairs (which is not PAX compliant) into a single
comma-delimited list of numbers (which is now PAX compliant).

Thus, we validate the following:
* The (offset, numbytes) headers must come in the correct order.
* The ',' delimiter cannot appear in the value.
We do not validate that the value is a parsible decimal since that
will be determined later.

Change-Id: I8d6681021734eb997898227ae8603efb1e17c0c8
Reviewed-on: https://go-review.googlesource.com/31439
Run-TryBot: Joe Tsai 
TryBot-Result: Gobot Gobot 
Reviewed-by: Brad Fitzpatrick

archive/tar: fix parsePAXTime

2016-10-20T01:06:09+00:00

Issues fixed:
* Could not handle quantity of seconds greater than 1<<31 on
32bit machines since strconv.ParseInt did not treat integers as 64b.
* Did not handle negative timestamps properly if nanoseconds were used.
Note that "-123.456" should result in a call to time.Unix(-123, -456000000).
* Incorrectly allowed a '-' right after the '.' (e.g., -123.-456)
* Did not detect invalid input after the truncation point (e.g., 123.123456789badbadbad).

Note that negative timestamps are allowed by PAX, but are not guaranteed
to be portable. See the relevant specification:
<<<
If pax encounters a file with a negative timestamp in copy or write mode,
it can reject the file, substitute a non-negative timestamp, or generate
a non-portable timestamp with a leading '-'.
>>>

Since the previous behavior already partially supported negative timestamps,
we are bound by Go's compatibility rules to keep support for them.
However, we should at least make sure we handle them properly.

Change-Id: I5686997708bfb59110ea7981175427290be737d1
Reviewed-on: https://go-review.googlesource.com/31441
Run-TryBot: Joe Tsai 
TryBot-Result: Gobot Gobot 
Reviewed-by: Brad Fitzpatrick

archive/tar: fix parsePAX to be POSIX.1-2001 compliant

2016-10-19T18:39:30+00:00

Relevant PAX specification:
<<<
If the  field is zero length, it shall delete any header
block field, previously entered extended header value, or
global extended header value of the same name.
>>>

We don't delete global extender headers since the Reader doesn't
even support global headers (which the specification admits was
a controversial feature).

Change-Id: I2125a5c907b23a3dc439507ca90fa5dc47d474a9
Reviewed-on: https://go-review.googlesource.com/31440
Run-TryBot: Brad Fitzpatrick 
TryBot-Result: Gobot Gobot 
Reviewed-by: Brad Fitzpatrick

archive/tar: compact slices in tests

2016-10-19T18:38:34+00:00

Took this opportunity to also embed tables in the functions
that they are actually used in and other stylistic cleanups.

There was no logical changes to the tests.

Change-Id: Ifa724060532175f6f4407d6cedc841891efd8f7b
Reviewed-on: https://go-review.googlesource.com/31436
Run-TryBot: Joe Tsai 
TryBot-Result: Gobot Gobot 
Reviewed-by: Brad Fitzpatrick

archive/tar: make Reader handle GNU format properly

2016-10-19T18:07:55+00:00

The GNU format does not have a prefix field, so we should make
no attempt to read it. It does however have atime and ctime fields.
Since Go previously placed incorrect values here, we liberally
read the atime and ctime fields and ignore errors so that old tar
files written by Go can at least be partially read.

This fixes half of #12594. The Writer is much harder to fix.

Updates #12594

Change-Id: Ia32845e2f262ee53366cf41dfa935f4d770c7a30
Reviewed-on: https://go-review.googlesource.com/31444
Run-TryBot: Joe Tsai 
TryBot-Result: Gobot Gobot 
Reviewed-by: Brad Fitzpatrick

archive/tar: reduce allocations in formatOctal

2016-10-12T21:27:51+00:00

Change-Id: I9ddb7d2a97d28aba7a107b65f278993daf7807fa
Reviewed-on: https://go-review.googlesource.com/30960
Reviewed-by: Brad Fitzpatrick 
Run-TryBot: Joe Tsai 
TryBot-Result: Gobot Gobot

archive/tar: fix and cleanup readOldGNUSparseMap

2016-10-12T16:36:57+00:00

* Assert that the format is GNU.
Both GNU and STAR have some form of sparse file support with
incompatible header structures. Worse yet, both formats use the
'S' type flag to indicate the presence of a sparse file.
As such, we should check the format (based on magic numbers)
and fail early.

* Move realsize parsing logic into readOldGNUSparseMap.
This is related to the sparse parsing logic and belongs here.

* Fix the termination condition for parsing sparse fields.
The termination condition for reading the sparse fields
is to simply check if the first byte of the offset field is NULL.
This does not seem to be documented in the GNU manual, but this is
the check done by the both the GNU and BSD implementations:
http://git.savannah.gnu.org/cgit/tar.git/tree/src/sparse.c?id=9a33077a7b7ad7d32815a21dee54eba63b38a81c#n731
https://github.com/libarchive/libarchive/blob/1fa9c7bf90f0862036a99896b0501c381584451a/libarchive/archive_read_support_format_tar.c#L2207

* Fix the parsing of sparse fields to use parseNumeric.
This is what GNU and BSD do. The previous two links show that
GNU and BSD both handle base-256 and base-8.

* Detect truncated streams.
The call to io.ReadFull does not check if the error is io.EOF.
Getting io.EOF in this method is never okay and should always be
converted to io.ErrUnexpectedEOF.

* Simplify the function.
The logic is essentially a do-while loop so we can remove
some redundant code.

Change-Id: Ib2f601b1a283eaec1e41b1d3396d649c80749c4e
Reviewed-on: https://go-review.googlesource.com/28471
Reviewed-by: Russ Cox
Run-TryBot: Russ Cox

archive/tar: handle integer overflow on 32bit machines

2016-10-12T16:36:41+00:00

Most calls to strconv.ParseInt(x, 10, 0) should really be
calls to strconv.ParseInt(x, 10, 64) in order to ensure that they
do not overflow on 32b architectures.

Furthermore, we should document a bug where Uid and Gid may
overflow on 32b machines since the type is declared as int.

Change-Id: I99c0670b3c2922e4a9806822d9ad37e1a364b2b8
Reviewed-on: https://go-review.googlesource.com/28472
Run-TryBot: Joe Tsai 
TryBot-Result: Gobot Gobot 
Reviewed-by: Russ Cox

archive/tar: move parse/format functionality into strconv.go

2016-09-29T18:38:28+00:00

Move all parse/format related functionality into strconv.go
and thoroughly test them. This also reduces the amount of noise
inside reader.go and writer.go.

There was zero functionality change other than moving code around.

Change-Id: I3bc288d10c20ebb3814b30b75d8acd7be62b85d7
Reviewed-on: https://go-review.googlesource.com/28470
Run-TryBot: Brad Fitzpatrick 
TryBot-Result: Gobot Gobot 
Reviewed-by: Brad Fitzpatrick