summaryrefslogtreecommitdiff
path: root/src/compress
Commit message (Collapse)AuthorAgeFilesLines
* compress/lzw: don't follow code == hi if last is invalid.Nigel Tao2017-06-082-1/+72
| | | | | | | | Fixes #20597. Change-Id: Iaa059277c02e80a2fbef59cf8bb3c58d2b90fda0 Reviewed-on: https://go-review.googlesource.com/45111 Reviewed-by: David Symonds <dsymonds@golang.org>
* compress/gzip: clarify behavior of Writer.CloseJoe Tsai2017-06-021-2/+3
| | | | | | | | Fixes #20551 Change-Id: Ia47cae14a26fe5f278ad7209218d083cc50a3ff8 Reviewed-on: https://go-review.googlesource.com/44572 Reviewed-by: Ian Lance Taylor <iant@golang.org>
* compress/bzip2: remove dead code in huffman.goJoe Tsai2017-05-241-4/+0
| | | | | | | | | | | | | | | | | The logic performs a series of shifts, which are useless given that they are followed by an assignment that overrides the value of the previous computation. I suspect (but cannot prove) that this is leftover logic from an original approach that attempted to store both the Huffman code and the length within the same variable instead of using two different variables as it currently does now. Fixes #17949 Change-Id: Ibf6c807c6cef3b28bfdaf2b68d9bc13503ac21b2 Reviewed-on: https://go-review.googlesource.com/44091 Reviewed-by: Ian Lance Taylor <iant@golang.org>
* compress/lzw: fix hi code overflow.Nigel Tao2017-04-282-1/+37
| | | | | | Change-Id: I2d3c3c715d857305944cd96c45554a16cb7967e9 Reviewed-on: https://go-review.googlesource.com/42032 Reviewed-by: David Symonds <dsymonds@golang.org>
* Revert "all: test adjustments for the iOS builder"Elias Naur2017-03-041-15/+9
| | | | | | | | | | | | This reverts commit 467109bf56fb560d1fd8a27c6184dbfe4f64ffef. Replaced by a improved strategy later in the CL relation chain. Change-Id: Ib90813b5a6c4716b563c8496013d2d57f9c022b8 Reviewed-on: https://go-review.googlesource.com/36066 Reviewed-by: David Crawshaw <crawshaw@golang.org> Run-TryBot: Elias Naur <elias.naur@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: use math/bits.Reverse8/16 instead of local implementationRobert Griesemer2017-03-013-51/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | No measurable impact on performance (specifically, no degradation). Reverse is used in Huffman en/de-coding. For completeness, here are all the speed-related benchmark results: name old time/op new time/op delta Decode/Digits/Huffman/1e4-8 181µs ± 0% 178µs ± 1% ~ (p=0.100 n=3+3) Decode/Digits/Huffman/1e5-8 1.60ms ± 3% 1.56ms ± 3% ~ (p=0.400 n=3+3) Decode/Digits/Huffman/1e6-8 15.7ms ± 1% 15.3ms ± 3% ~ (p=0.700 n=3+3) Decode/Digits/Speed/1e4-8 179µs ± 0% 180µs ± 0% ~ (p=0.200 n=3+3) Decode/Digits/Speed/1e5-8 1.68ms ± 0% 1.66ms ± 3% ~ (p=0.700 n=3+3) Decode/Digits/Speed/1e6-8 16.6ms ± 2% 16.6ms ± 5% ~ (p=0.700 n=3+3) Decode/Digits/Default/1e4-8 179µs ± 1% 178µs ± 1% ~ (p=0.700 n=3+3) Decode/Digits/Default/1e5-8 1.62ms ± 3% 1.62ms ± 4% ~ (p=1.000 n=3+3) Decode/Digits/Default/1e6-8 16.0ms ± 2% 16.0ms ± 3% ~ (p=1.000 n=3+3) Decode/Digits/Compression/1e4-8 179µs ± 1% 179µs ± 0% ~ (p=0.200 n=3+3) Decode/Digits/Compression/1e5-8 1.62ms ± 2% 1.62ms ± 3% ~ (p=1.000 n=3+3) Decode/Digits/Compression/1e6-8 16.1ms ± 3% 16.0ms ± 3% ~ (p=1.000 n=3+3) Decode/Twain/Huffman/1e4-8 205µs ± 2% 207µs ± 1% ~ (p=1.000 n=3+3) Decode/Twain/Huffman/1e5-8 1.77ms ± 2% 1.77ms ± 4% ~ (p=0.700 n=3+3) Decode/Twain/Huffman/1e6-8 17.4ms ± 2% 17.4ms ± 3% ~ (p=1.000 n=3+3) Decode/Twain/Speed/1e4-8 186µs ± 1% 186µs ± 1% ~ (p=0.400 n=3+3) Decode/Twain/Speed/1e5-8 1.53ms ± 2% 1.52ms ± 0% ~ (p=0.700 n=3+3) Decode/Twain/Speed/1e6-8 14.9ms ± 1% 14.8ms ± 1% ~ (p=1.000 n=3+3) Decode/Twain/Default/1e4-8 176µs ± 1% 174µs ± 0% ~ (p=0.200 n=3+3) Decode/Twain/Default/1e5-8 1.30ms ± 2% 1.31ms ± 1% ~ (p=0.700 n=3+3) Decode/Twain/Default/1e6-8 12.6ms ± 3% 12.5ms ± 0% ~ (p=0.700 n=3+3) Decode/Twain/Compression/1e4-8 177µs ± 0% 174µs ± 1% ~ (p=0.100 n=3+3) Decode/Twain/Compression/1e5-8 1.30ms ± 1% 1.31ms ± 0% ~ (p=0.700 n=3+3) Decode/Twain/Compression/1e6-8 12.5ms ± 1% 12.5ms ± 1% ~ (p=1.000 n=3+3) Encode/Digits/Huffman/1e4-8 47.4µs ± 1% 46.5µs ± 0% ~ (p=0.100 n=3+3) Encode/Digits/Huffman/1e5-8 453µs ± 2% 446µs ± 1% ~ (p=0.700 n=3+3) Encode/Digits/Huffman/1e6-8 4.44ms ± 3% 4.39ms ± 0% ~ (p=1.000 n=3+3) Encode/Digits/Speed/1e4-8 190µs ± 4% 185µs ± 0% ~ (p=0.100 n=3+3) Encode/Digits/Speed/1e5-8 1.78ms ± 5% 1.75ms ± 1% ~ (p=1.000 n=3+3) Encode/Digits/Speed/1e6-8 17.9ms ± 7% 17.3ms ± 1% ~ (p=0.400 n=3+3) Encode/Digits/Default/1e4-8 366µs ± 1% 361µs ± 0% ~ (p=0.200 n=3+3) Encode/Digits/Default/1e5-8 5.58ms ± 5% 5.44ms ± 1% ~ (p=0.400 n=3+3) Encode/Digits/Default/1e6-8 59.0ms ± 3% 58.2ms ± 1% ~ (p=0.700 n=3+3) Encode/Digits/Compression/1e4-8 369µs ± 3% 362µs ± 0% ~ (p=0.100 n=3+3) Encode/Digits/Compression/1e5-8 5.50ms ± 2% 5.47ms ± 1% ~ (p=1.000 n=3+3) Encode/Digits/Compression/1e6-8 59.4ms ± 2% 58.5ms ± 1% ~ (p=0.400 n=3+3) Encode/Twain/Huffman/1e4-8 64.4µs ± 3% 64.7µs ± 1% ~ (p=0.700 n=3+3) Encode/Twain/Huffman/1e5-8 526µs ± 1% 526µs ± 2% ~ (p=1.000 n=3+3) Encode/Twain/Huffman/1e6-8 5.18ms ± 2% 5.17ms ± 1% ~ (p=0.700 n=3+3) Encode/Twain/Speed/1e4-8 206µs ± 1% 204µs ± 0% ~ (p=0.100 n=3+3) Encode/Twain/Speed/1e5-8 1.73ms ± 2% 1.70ms ± 0% ~ (p=0.100 n=3+3) Encode/Twain/Speed/1e6-8 16.7ms ± 0% 16.7ms ± 1% ~ (p=1.000 n=3+3) Encode/Twain/Default/1e4-8 423µs ± 3% 418µs ± 1% ~ (p=1.000 n=3+3) Encode/Twain/Default/1e5-8 6.34ms ± 4% 6.23ms ± 0% ~ (p=1.000 n=3+3) Encode/Twain/Default/1e6-8 68.0ms ± 3% 67.5ms ± 0% ~ (p=0.700 n=3+3) Encode/Twain/Compression/1e4-8 435µs ± 3% 424µs ± 0% ~ (p=0.700 n=3+3) Encode/Twain/Compression/1e5-8 7.01ms ± 1% 6.92ms ± 0% ~ (p=0.100 n=3+3) Encode/Twain/Compression/1e6-8 77.1ms ± 4% 75.5ms ± 1% ~ (p=0.400 n=3+3) name old speed new speed delta Decode/Digits/Huffman/1e4-8 55.2MB/s ± 0% 56.2MB/s ± 1% ~ (p=0.100 n=3+3) Decode/Digits/Huffman/1e5-8 62.4MB/s ± 3% 64.1MB/s ± 3% ~ (p=0.400 n=3+3) Decode/Digits/Huffman/1e6-8 63.8MB/s ± 1% 65.3MB/s ± 3% ~ (p=0.700 n=3+3) Decode/Digits/Speed/1e4-8 55.8MB/s ± 0% 55.4MB/s ± 0% ~ (p=0.200 n=3+3) Decode/Digits/Speed/1e5-8 59.6MB/s ± 0% 60.3MB/s ± 3% ~ (p=0.700 n=3+3) Decode/Digits/Speed/1e6-8 60.1MB/s ± 2% 60.3MB/s ± 4% ~ (p=0.700 n=3+3) Decode/Digits/Default/1e4-8 55.8MB/s ± 1% 56.1MB/s ± 1% ~ (p=0.700 n=3+3) Decode/Digits/Default/1e5-8 61.8MB/s ± 3% 61.7MB/s ± 4% ~ (p=1.000 n=3+3) Decode/Digits/Default/1e6-8 62.4MB/s ± 2% 62.4MB/s ± 3% ~ (p=1.000 n=3+3) Decode/Digits/Compression/1e4-8 55.7MB/s ± 1% 56.0MB/s ± 0% ~ (p=0.300 n=3+3) Decode/Digits/Compression/1e5-8 61.7MB/s ± 2% 61.9MB/s ± 3% ~ (p=1.000 n=3+3) Decode/Digits/Compression/1e6-8 62.2MB/s ± 3% 62.6MB/s ± 3% ~ (p=1.000 n=3+3) Decode/Twain/Huffman/1e4-8 48.8MB/s ± 2% 48.4MB/s ± 1% ~ (p=1.000 n=3+3) Decode/Twain/Huffman/1e5-8 56.4MB/s ± 2% 56.6MB/s ± 4% ~ (p=0.700 n=3+3) Decode/Twain/Huffman/1e6-8 57.6MB/s ± 2% 57.5MB/s ± 3% ~ (p=1.000 n=3+3) Decode/Twain/Speed/1e4-8 53.7MB/s ± 1% 53.9MB/s ± 1% ~ (p=0.400 n=3+3) Decode/Twain/Speed/1e5-8 65.5MB/s ± 2% 65.6MB/s ± 0% ~ (p=0.700 n=3+3) Decode/Twain/Speed/1e6-8 66.9MB/s ± 1% 67.4MB/s ± 1% ~ (p=1.000 n=3+3) Decode/Twain/Default/1e4-8 56.9MB/s ± 1% 57.3MB/s ± 0% ~ (p=0.200 n=3+3) Decode/Twain/Default/1e5-8 77.2MB/s ± 2% 76.6MB/s ± 1% ~ (p=0.700 n=3+3) Decode/Twain/Default/1e6-8 79.3MB/s ± 3% 80.0MB/s ± 0% ~ (p=0.700 n=3+3) Decode/Twain/Compression/1e4-8 56.4MB/s ± 0% 57.5MB/s ± 1% ~ (p=0.100 n=3+3) Decode/Twain/Compression/1e5-8 76.8MB/s ± 1% 76.5MB/s ± 0% ~ (p=0.700 n=3+3) Decode/Twain/Compression/1e6-8 80.1MB/s ± 1% 79.8MB/s ± 1% ~ (p=1.000 n=3+3) Encode/Digits/Huffman/1e4-8 211MB/s ± 1% 215MB/s ± 0% ~ (p=0.100 n=3+3) Encode/Digits/Huffman/1e5-8 221MB/s ± 2% 224MB/s ± 1% ~ (p=0.700 n=3+3) Encode/Digits/Huffman/1e6-8 225MB/s ± 3% 228MB/s ± 0% ~ (p=1.000 n=3+3) Encode/Digits/Speed/1e4-8 52.8MB/s ± 4% 54.1MB/s ± 0% ~ (p=0.100 n=3+3) Encode/Digits/Speed/1e5-8 56.2MB/s ± 5% 57.0MB/s ± 1% ~ (p=1.000 n=3+3) Encode/Digits/Speed/1e6-8 56.0MB/s ± 6% 57.7MB/s ± 1% ~ (p=0.400 n=3+3) Encode/Digits/Default/1e4-8 27.3MB/s ± 1% 27.7MB/s ± 0% ~ (p=0.200 n=3+3) Encode/Digits/Default/1e5-8 17.9MB/s ± 4% 18.4MB/s ± 1% ~ (p=0.400 n=3+3) Encode/Digits/Default/1e6-8 17.0MB/s ± 3% 17.2MB/s ± 1% ~ (p=0.500 n=3+3) Encode/Digits/Compression/1e4-8 27.1MB/s ± 3% 27.6MB/s ± 0% ~ (p=0.100 n=3+3) Encode/Digits/Compression/1e5-8 18.2MB/s ± 2% 18.3MB/s ± 1% ~ (p=1.000 n=3+3) Encode/Digits/Compression/1e6-8 16.9MB/s ± 2% 17.1MB/s ± 1% ~ (p=0.400 n=3+3) Encode/Twain/Huffman/1e4-8 155MB/s ± 3% 155MB/s ± 1% ~ (p=0.700 n=3+3) Encode/Twain/Huffman/1e5-8 190MB/s ± 1% 190MB/s ± 2% ~ (p=1.000 n=3+3) Encode/Twain/Huffman/1e6-8 193MB/s ± 2% 193MB/s ± 1% ~ (p=0.700 n=3+3) Encode/Twain/Speed/1e4-8 48.5MB/s ± 1% 49.1MB/s ± 0% ~ (p=0.100 n=3+3) Encode/Twain/Speed/1e5-8 57.7MB/s ± 2% 59.0MB/s ± 0% ~ (p=0.100 n=3+3) Encode/Twain/Speed/1e6-8 59.7MB/s ± 0% 59.7MB/s ± 1% ~ (p=1.000 n=3+3) Encode/Twain/Default/1e4-8 23.6MB/s ± 3% 23.9MB/s ± 1% ~ (p=1.000 n=3+3) Encode/Twain/Default/1e5-8 15.8MB/s ± 4% 16.1MB/s ± 0% ~ (p=1.000 n=3+3) Encode/Twain/Default/1e6-8 14.7MB/s ± 3% 14.8MB/s ± 0% ~ (p=0.700 n=3+3) Encode/Twain/Compression/1e4-8 23.0MB/s ± 3% 23.6MB/s ± 0% ~ (p=0.700 n=3+3) Encode/Twain/Compression/1e5-8 14.3MB/s ± 1% 14.5MB/s ± 0% ~ (p=0.100 n=3+3) Encode/Twain/Compression/1e6-8 13.0MB/s ± 4% 13.2MB/s ± 1% ~ (p=0.400 n=3+3) Measured on a "quiet" (no browser running) 2.3 GHz Intel Core i7, running macOS 10.12.3. See also #19279. Change-Id: Ice759eb34eb37442b543957447c264e0aadc1fa9 Reviewed-on: https://go-review.googlesource.com/37460 Run-TryBot: Robert Griesemer <gri@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* all: test adjustments for the iOS builderDavid Crawshaw2017-01-141-9/+15
| | | | | | | | | | | | | | | | | | | | The working directory is now adjusted to match the typical Go test working directory in main, as the old trick for adjusting earlier stopped working with the latest version of LLDB bugs. That means the small number of places where testdata files are read before main is called no longer work. This CL adjusts those reads to happen after main is called. (This has the bonus effect of not reading some benchmark testdata files in all.bash.) Fixes compress/bzip2, go/doc, go/parser, os, and time package tests on the iOS builder. Change-Id: If60f026aa7848b37511c36ac5e3985469ec25209 Reviewed-on: https://go-review.googlesource.com/35255 Run-TryBot: David Crawshaw <crawshaw@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: avoid large stack growth in fillDeflateJoe Tsai2017-01-123-4/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ranging over an array causes the array to be copied over to the stack, which cause large re-growths. Instead, we should iterate over slices of the array. Also, assigning a large struct literal uses the stack even though the actual fields being populated are small in comparison to the entirety of the struct (see #18636). Fixing the stack growth does not alter CPU-time performance much since the stack-growth and copying was such a tiny portion of the compression work: name old time/op new time/op delta Encode/Digits/Default/1e4-8 332µs ± 1% 332µs ± 1% ~ (p=0.796 n=10+10) Encode/Digits/Default/1e5-8 5.07ms ± 2% 5.05ms ± 1% ~ (p=0.815 n=9+8) Encode/Digits/Default/1e6-8 53.7ms ± 1% 53.9ms ± 1% ~ (p=0.075 n=10+10) Encode/Twain/Default/1e4-8 380µs ± 1% 380µs ± 1% ~ (p=0.684 n=10+10) Encode/Twain/Default/1e5-8 5.79ms ± 2% 5.79ms ± 1% ~ (p=0.497 n=9+10) Encode/Twain/Default/1e6-8 61.5ms ± 1% 61.8ms ± 1% ~ (p=0.247 n=10+10) name old speed new speed delta Encode/Digits/Default/1e4-8 30.1MB/s ± 1% 30.1MB/s ± 1% ~ (p=0.753 n=10+10) Encode/Digits/Default/1e5-8 19.7MB/s ± 2% 19.8MB/s ± 1% ~ (p=0.795 n=9+8) Encode/Digits/Default/1e6-8 18.6MB/s ± 1% 18.5MB/s ± 1% ~ (p=0.072 n=10+10) Encode/Twain/Default/1e4-8 26.3MB/s ± 1% 26.3MB/s ± 1% ~ (p=0.616 n=10+10) Encode/Twain/Default/1e5-8 17.3MB/s ± 2% 17.3MB/s ± 1% ~ (p=0.484 n=9+10) Encode/Twain/Default/1e6-8 16.3MB/s ± 1% 16.2MB/s ± 1% ~ (p=0.238 n=10+10) Updates #18636 Fixes #18625 Change-Id: I471b20339bf675f63dc56d38b3acdd824fe23328 Reviewed-on: https://go-review.googlesource.com/35122 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/gzip: skip TestGZIPFilesHaveZeroMTimes on non-buildersJoe Tsai2017-01-111-4/+10
| | | | | | | | | | Fixes #18604 Change-Id: I89221d5e632042167dfced068e1dc14e932cd618 Reviewed-on: https://go-review.googlesource.com/35111 Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* compress/flate: add examplesJoe Tsai2016-11-121-0/+243
| | | | | | | | | | Updates #16360 Change-Id: I66ff23e0501363f58fe891d5e95806422071f93b Reviewed-on: https://go-review.googlesource.com/30162 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* all: sprinkle t.Parallel on some slow testsBrad Fitzpatrick2016-11-043-0/+11
| | | | | | | | | | | | | | | | | I used the slowtests.go tool as described in https://golang.org/cl/32684 on packages that stood out. go test -short std drops from ~56 to ~52 seconds. This isn't a huge win, but it was mostly an exercise. Updates #17751 Change-Id: I9f3402e36a038d71e662d06ce2c1d52f6c4b674d Reviewed-on: https://go-review.googlesource.com/32751 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
* compress/flate: tighten the BestSpeed max match offset bound.Nigel Tao2016-10-302-5/+62
| | | | | | | | | | | | Previously, we were off by one. Also fix a comment typo. Change-Id: Ib94d23acc56d5fccd44144f71655481f98803ac8 Reviewed-on: https://go-review.googlesource.com/32149 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/gzip: only encode MTIME if it is validJoe Tsai2016-10-294-11/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | The GZIP format records the ModTime as an uint32 counting seconds since the Unix epoch. The zero value is explicitly defined in section 2.3.1 as meaning no timestamp is available. Currently, the Writer always encodes the ModTime even if it is the zero time.Time value, which causes the Writer to try and encode the value -62135596800 into the uint32 MTIME field. This causes an overflow and results in our GZIP files having MTIME fields indicating a date in 2042-07-13. We alter the Writer to only encode ModTime if the value does not underflow the MTIME field (i.e., it is newer than the Unix epoch). We do not attempt to fix what happens when the timestamp overflows in the year 2106. We alter the Reader to only decode ModTime if the value is non-zero. There is no risk of overflowing time.Time when decoding. Fixes #17663 Change-Id: Ie1b65770c6342cd7b14aeebe10e5a49e6c9eb730 Reviewed-on: https://go-review.googlesource.com/32325 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: make compression level 0 consistentKlaus Post2016-10-272-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | Tests for determinism was not working as intended since io.Copybuffer uses the io.WriterTo if available. This exposed that level 0 (no compression) changed output based on the number of writes and buffers given to the writer. Previously, Write would emit a new raw block (BTYPE=00) for every non-empty call to Write. This CL fixes it such that a raw block is only emitted upon the following conditions: * A full window is obtained (every 65535 bytes) * Flush is called * Close is called Change-Id: I807f866d97e2db7820f11febab30a96266a6cbf1 Reviewed-on: https://go-review.googlesource.com/31174 Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
* compress/flate: level 1 (best speed) match across blocksKlaus Post2016-10-273-48/+267
| | | | | | | | | | | | | | | | | | | | | This change makes deflate level 1 (best speed) match across block boundaries. This comes at a small speed penalty, but improves compression on almost all output. Sample numbers on various content types: enwik9: 391052014 -> 382578469 bytes, 77.59 -> 74.28 MB/s adresser.001: 57269799 -> 47756095 bytes, 287.84 -> 357.86 MB/s 10gb: 5233055166 -> 5198328382 bytes, 105.85 -> 96.99 MB/s rawstudio-mint14: 3972329211 -> 3927423364 bytes, 100.07 -> 94.22 MB/s sites: 165556800 -> 163178702 bytes, 72.31 -> 70.15 MB/s objectfiles: 115962472 -> 111649524 bytes, 132.60 -> 128.05 MB/s sharnd.out: 200015283 -> 200015283 bytes, 221.50 -> 218.83 MB/s Change-Id: I62a139e5c06976e803439a4268acede5139b8cfc Reviewed-on: https://go-review.googlesource.com/31640 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Reviewed-by: Nigel Tao <nigeltao@golang.org>
* all: minor vet fixesJosh Bleecher Snyder2016-10-241-1/+1
| | | | | | | | Change-Id: I22f0f3e792052762499f632571155768b4052bc9 Reviewed-on: https://go-review.googlesource.com/31759 Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* compress/gzip, compress/zlib: add HuffmanOnly as compression levels.Klaus Post2016-10-203-9/+16
| | | | | | | | | | | | This exposes HuffmanOnly in zlib and gzip packages, which is currently unavailable. Change-Id: If5d103bbc8b5fce2f5d740fd103a235c5d1ed7cd Reviewed-on: https://go-review.googlesource.com/31186 Reviewed-by: Nigel Tao <nigeltao@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: use correct table for size estimationKlaus Post2016-10-171-1/+1
| | | | | | | | | | | The incorrect table was used for estimating output size. This can give suboptimal selection of entropy encoder in rare cases. Change-Id: I8b358200f2d1f9a3f9b79a44269d7be704e1d2d9 Reviewed-on: https://go-review.googlesource.com/31172 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/gzip: document Reader.ReadEmmanuel Odeke2016-10-091-0/+1
| | | | | | | | | Fixes #17374. Change-Id: Ic89c35aaa31f35a8a4e3ffa09f49b68f08127625 Reviewed-on: https://go-review.googlesource.com/30718 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
* compress/gzip: add examplesEmmanuel Odeke2016-09-221-0/+128
| | | | | | | | | | | | | | | | | Updates #16360. Adds examples uing: + Writer, Reader + Reader.Multistream to concatenate and then individually retrieve multiple gzipped files + Reset Change-Id: I9ad9b92729a5cd58f7368eaf2db05f1cdf21063d Reviewed-on: https://go-review.googlesource.com/29218 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: clarify the behavior of Writer.FlushJoe Tsai2016-09-041-1/+3
| | | | | | | | | | Fixes #16068 Change-Id: I04e80a181c0b7356996f7a1158ea4895ff9e1e39 Reviewed-on: https://go-review.googlesource.com/28477 Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* compress/flate: always return uncompressed data in the event of errorJoe Tsai2016-08-314-0/+70
| | | | | | | | | | | | | In the event of an unexpected error, we should always flush available decompressed data to the user. Fixes #16924 Change-Id: I0bc0824c3201f3149e84e6a26e3dbcba72a1aae5 Reviewed-on: https://go-review.googlesource.com/28216 Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
* compress/flate: make huffmanBitWriter errors persistentJoe Tsai2016-08-203-14/+78
| | | | | | | | | | | | | | | | | | | | | | For persistent error handling, the methods of huffmanBitWriter have to be consistent about how they check errors. It must either consistently check error *before* every operation OR immediately *after* every operation. Since most of the current logic uses the previous approach, we apply the same style of error checking to writeBits and all calls to Write such that they only operate if w.err is already nil going into them. The error handling approach is brittle and easily broken by future commits to the code. In the near future, we should switch the logic to use panic at the lowest levels and a recover at the edge of the public API to ensure that errors are always persistent. Fixes #16749 Change-Id: Ie1d83e4ed8842f6911a31e23311cd3cbf38abe8c Reviewed-on: https://go-review.googlesource.com/27200 Reviewed-by: Matthew Dempsky <mdempsky@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* compress/flate: document HuffmanOnlyJoe Tsai2016-07-251-2/+11
| | | | | | | | | | Fixes #16489 Change-Id: I13e2ed6de59102f977566de637d8d09b4e541980 Reviewed-on: https://go-review.googlesource.com/25200 Reviewed-by: Andrew Gerrand <adg@golang.org> Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: don't ignore dict in Reader.ResetVladimir Mihailenco2016-06-272-1/+31
| | | | | | | | | Fixes #16162. Change-Id: I6f4ae906630079ef5fc29ee5f70e2e3d1c962170 Reviewed-on: https://go-review.googlesource.com/24390 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: use seperate const block for exported constantsJoe Tsai2016-05-311-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | As rendered on https://tip.golang.org/pkg/compress/flate/, there is an extra new-line because of the unexported constants in the same block. <<< const ( NoCompression = 0 BestSpeed = 1 BestCompression = 9 DefaultCompression = -1 HuffmanOnly = -2 // Disables match search and only does Huffman entropy reduction. ) >>> Instead, seperate the exported compression level constants into its own const block. This is both more readable and also fixes the issue. Change-Id: I60b7966c83fb53356c02e4640d05f55a3bee35b7 Reviewed-on: https://go-review.googlesource.com/23557 Reviewed-by: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: simplify using subtests and sub-benchmarksMarcel van Lohuizen2016-05-252-133/+94
| | | | | | | | | | | | | | | | | This causes the large files to be loaded only once per benchmark. This CL also serves as an example use case of sub(tests|-benchmarks). This CL ensures that names are identical to the original except for an added slashes. Things could be simplified further if this restriction were dropped. Change-Id: I45e303e158e3152e33d0d751adfef784713bf997 Reviewed-on: https://go-review.googlesource.com/23420 Reviewed-by: Russ Cox <rsc@golang.org> Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/lzw: use Run for benchmarksMarcel van Lohuizen2016-05-252-62/+54
| | | | | | | | | | load file only once per group. Change-Id: I965661507055e6e100506bf14d37133ecdd2cc5e Reviewed-on: https://go-review.googlesource.com/23423 Reviewed-by: Russ Cox <rsc@golang.org> Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress: update documentation regarding footer verificationJoe Tsai2016-05-092-1/+6
| | | | | | | | | | | | | | | Address two documentation issues: 1) Document that the GZIP and ZLIB footer is only verified when the reader has been fully consumed. 2) The zlib reader is guaranteed to not read past the EOF if the input io.Reader is also a io.ByteReader. This functionality was documented in the flate and gzip packages but not on zlib. Fixes #14867 Change-Id: I43d46b93e38f98a04901dc7d4f18ed2f9e09f6fb Reviewed-on: https://go-review.googlesource.com/21218 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* compress/flate: distinguish between base and min match length.Nigel Tao2016-05-053-17/+24
| | | | | | Change-Id: I93db5cd86e3fb568e4444cad95268ba4a02ce8a0 Reviewed-on: https://go-review.googlesource.com/22787 Reviewed-by: Nigel Tao <nigeltao@golang.org>
* compress/flate: use a constant hash table size for Best Speed.Nigel Tao2016-04-291-24/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes compress/flate's version of Snappy diverge from the upstream golang/snappy version, but the latter has a goal of matching C++ snappy output byte-for-byte. Both C++ and the asm version of golang/snappy can use a smaller N for the O(N) zero-initialization of the hash table when the input is small, even if the pure Go golang/snappy algorithm cannot: "var table [tableSize]uint16" zeroes all tableSize elements. For this package, we don't have the match-C++-snappy goal, so we can use a different (constant) hash table size. This is a small win, in terms of throughput and output size, but it also enables us to re-use the (constant size) hash table between encodeBestSpeed calls, avoiding the cost of zero-initializing the hash table altogether. This will be implemented in follow-up commits. This package's benchmarks: name old speed new speed delta EncodeDigitsSpeed1e4-8 72.8MB/s ± 1% 73.5MB/s ± 1% +0.86% (p=0.000 n=10+10) EncodeDigitsSpeed1e5-8 77.5MB/s ± 1% 78.0MB/s ± 0% +0.69% (p=0.000 n=10+10) EncodeDigitsSpeed1e6-8 82.0MB/s ± 1% 82.7MB/s ± 1% +0.85% (p=0.000 n=10+9) EncodeTwainSpeed1e4-8 65.1MB/s ± 1% 65.6MB/s ± 0% +0.78% (p=0.000 n=10+9) EncodeTwainSpeed1e5-8 80.0MB/s ± 0% 80.6MB/s ± 1% +0.66% (p=0.000 n=9+10) EncodeTwainSpeed1e6-8 81.6MB/s ± 1% 82.1MB/s ± 1% +0.55% (p=0.017 n=10+10) Input size in bytes, output size (and time taken) before and after on some larger files: 1073741824 57269781 ( 3183ms) 57269781 ( 3177ms) adresser.001 1000000000 391052000 ( 11071ms) 391051996 ( 11067ms) enwik9 1911399616 378679516 ( 13450ms) 378679514 ( 13079ms) gob-stream 8558382592 3972329193 ( 99962ms) 3972329193 ( 91290ms) rawstudio-mint14.tar 200000000 200015265 ( 776ms) 200015265 ( 774ms) sharnd.out Thanks to Klaus Post for the original suggestion on cl/21021. Change-Id: Ia4c63a8d1b92c67e1765ec5c3c8c69d289d9a6ce Reviewed-on: https://go-review.googlesource.com/22604 Reviewed-by: Russ Cox <rsc@golang.org>
* compress/flate: replace "Best Speed" with specialized versionNigel Tao2016-04-293-9/+317
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This encoding algorithm, which prioritizes speed over output size, is based on Snappy's LZ77-style encoder: github.com/golang/snappy This commit keeps the diff between this package's encodeBestSpeed function and and Snappy's encodeBlock function as small as possible (see the diff below). Follow-up commits will improve this package's performance and output size. This package's speed benchmarks: name old speed new speed delta EncodeDigitsSpeed1e4-8 40.7MB/s ± 0% 73.0MB/s ± 0% +79.18% (p=0.008 n=5+5) EncodeDigitsSpeed1e5-8 33.0MB/s ± 0% 77.3MB/s ± 1% +134.04% (p=0.008 n=5+5) EncodeDigitsSpeed1e6-8 32.1MB/s ± 0% 82.1MB/s ± 0% +156.18% (p=0.008 n=5+5) EncodeTwainSpeed1e4-8 42.1MB/s ± 0% 65.0MB/s ± 0% +54.61% (p=0.008 n=5+5) EncodeTwainSpeed1e5-8 46.3MB/s ± 0% 80.0MB/s ± 0% +72.81% (p=0.008 n=5+5) EncodeTwainSpeed1e6-8 47.3MB/s ± 0% 81.7MB/s ± 0% +72.86% (p=0.008 n=5+5) Here's the milliseconds taken, before and after this commit, to compress a number of test files: Go's src/compress/testdata files: 4 1 e.txt 8 4 Mark.Twain-Tom.Sawyer.txt github.com/golang/snappy's benchmark files: 3 1 alice29.txt 12 3 asyoulik.txt 6 1 fireworks.jpeg 1 1 geo.protodata 1 0 html 2 2 html_x_4 6 3 kppkn.gtb 11 4 lcet10.txt 5 1 paper-100k.pdf 14 6 plrabn12.txt 17 6 urls.10K Larger files linked to from https://docs.google.com/spreadsheets/d/1VLxi-ac0BAtf735HyH3c1xRulbkYYUkFecKdLPH7NIQ/edit#gid=166102500 2409 3182 adresser.001 16757 11027 enwik9 13764 12946 gob-stream 153978 74317 rawstudio-mint14.tar 4371 770 sharnd.out Output size is larger. In the table below, the first column is the input size, the second column is the output size prior to this commit, the third column is the output size after this commit. 100003 47707 50006 e.txt 387851 172707 182930 Mark.Twain-Tom.Sawyer.txt 152089 62457 66705 alice29.txt 125179 54503 57274 asyoulik.txt 123093 122827 123108 fireworks.jpeg 118588 18574 20558 geo.protodata 102400 16601 17305 html 409600 65506 70313 html_x_4 184320 49007 50944 kppkn.gtb 426754 166957 179355 lcet10.txt 102400 82126 84937 paper-100k.pdf 481861 218617 231988 plrabn12.txt 702087 241774 258020 urls.10K 1073741824 43074110 57269781 adresser.001 1000000000 365772256 391052000 enwik9 1911399616 340364558 378679516 gob-stream 8558382592 3807229562 3972329193 rawstudio-mint14.tar 200000000 200061040 200015265 sharnd.out The diff between github.com/golang/snappy's encodeBlock function and this commit's encodeBestSpeed function: 1c1,7 < func encodeBlock(dst, src []byte) (d int) { --- > func encodeBestSpeed(dst []token, src []byte) []token { > // This check isn't in the Snappy implementation, but there, the caller > // instead of the callee handles this case. > if len(src) < minNonLiteralBlockSize { > return emitLiteral(dst, src) > } > 4c10 < // and len(src) <= maxBlockSize and maxBlockSize == 65536. --- > // and len(src) <= maxStoreBlockSize and maxStoreBlockSize == 65535. 65c71 < if load32(src, s) == load32(src, candidate) { --- > if s-candidate < maxOffset && load32(src, s) == load32(src, candidate) { 73c79 < d += emitLiteral(dst[d:], src[nextEmit:s]) --- > dst = emitLiteral(dst, src[nextEmit:s]) 90c96 < // This is an inlined version of: --- > // This is an inlined version of Snappy's: 93c99,103 < for i := candidate + 4; s < len(src) && src[i] == src[s]; i, s = i+1, s+1 { --- > s1 := base + maxMatchLength > if s1 > len(src) { > s1 = len(src) > } > for i := candidate + 4; s < s1 && src[i] == src[s]; i, s = i+1, s+1 { 96c106,107 < d += emitCopy(dst[d:], base-candidate, s-base) --- > // matchToken is flate's equivalent of Snappy's emitCopy. > dst = append(dst, matchToken(uint32(s-base-3), uint32(base-candidate-minOffsetSize))) 114c125 < if uint32(x>>8) != load32(src, candidate) { --- > if s-candidate >= maxOffset || uint32(x>>8) != load32(src, candidate) { 124c135 < d += emitLiteral(dst[d:], src[nextEmit:]) --- > dst = emitLiteral(dst, src[nextEmit:]) 126c137 < return d --- > return dst This change is based on https://go-review.googlesource.com/#/c/21021/ by Klaus Post, but it is a separate changelist as cl/21021 seems to have stalled in code review, and the Go 1.7 feature freeze approaches. Golang-dev discussion: https://groups.google.com/d/topic/golang-dev/XYgHX9p8IOk/discussion and of course cl/21021. Change-Id: Ib662439417b3bd0b61c2977c12c658db3e44d164 Reviewed-on: https://go-review.googlesource.com/22370 Reviewed-by: Russ Cox <rsc@golang.org>
* compress/flate: use uncompressed if dynamic encoding is largerKlaus Post2016-04-183-59/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds size calculation to "dynamic" writes. This ensures that if dynamic Huffman encoding is bigger, or only slightly smaller than raw data, the block is written uncompressed. To minimize the code duplication of this function, the size calculation has been moved to separate functions. Since I was modifying these calculations, I changed "int64" size calculations to "int". Blocks are of very limited size, so there is not any risk of overflows. This should mainly improve 32 bit performance, but amd64 also gets a slight boost: name old time/op new time/op delta EncodeDigitsHuffman1e4-8 49.9µs ± 1% 49.3µs ± 1% -1.21% (p=0.000 n=10+10) EncodeDigitsHuffman1e5-8 476µs ± 1% 471µs ± 3% ~ (p=0.218 n=10+10) EncodeDigitsHuffman1e6-8 4.80ms ± 2% 4.75ms ± 2% ~ (p=0.243 n=10+9) EncodeDigitsSpeed1e4-8 305µs ± 3% 300µs ± 1% -1.86% (p=0.005 n=10+10) EncodeDigitsSpeed1e5-8 3.67ms ± 2% 3.58ms ± 1% -2.29% (p=0.000 n=9+8) EncodeDigitsSpeed1e6-8 38.3ms ± 2% 37.0ms ± 1% -3.45% (p=0.000 n=9+9) EncodeDigitsDefault1e4-8 361µs ± 2% 353µs ± 1% -2.21% (p=0.000 n=10+10) EncodeDigitsDefault1e5-8 5.24ms ± 2% 5.19ms ± 2% ~ (p=0.105 n=10+10) EncodeDigitsDefault1e6-8 56.5ms ± 3% 55.1ms ± 1% -2.42% (p=0.001 n=10+10) EncodeDigitsCompress1e4-8 362µs ± 2% 358µs ± 2% ~ (p=0.123 n=10+10) EncodeDigitsCompress1e5-8 5.26ms ± 3% 5.20ms ± 1% ~ (p=0.089 n=10+10) EncodeDigitsCompress1e6-8 56.0ms ± 4% 55.0ms ± 1% ~ (p=0.065 n=10+9) EncodeTwainHuffman1e4-8 70.9µs ± 3% 67.6µs ± 2% -4.59% (p=0.000 n=10+10) EncodeTwainHuffman1e5-8 556µs ± 2% 533µs ± 1% -4.20% (p=0.000 n=10+10) EncodeTwainHuffman1e6-8 5.54ms ± 3% 5.29ms ± 1% -4.37% (p=0.000 n=10+9) EncodeTwainSpeed1e4-8 294µs ± 3% 293µs ± 1% ~ (p=0.965 n=10+8) EncodeTwainSpeed1e5-8 2.59ms ± 2% 2.56ms ± 1% ~ (p=0.353 n=10+10) EncodeTwainSpeed1e6-8 25.6ms ± 1% 24.9ms ± 1% -2.62% (p=0.000 n=9+10) EncodeTwainDefault1e4-8 419µs ± 2% 417µs ± 1% ~ (p=0.780 n=10+9) EncodeTwainDefault1e5-8 6.23ms ± 4% 6.16ms ± 1% ~ (p=0.218 n=10+10) EncodeTwainDefault1e6-8 66.2ms ± 2% 65.7ms ± 1% ~ (p=0.529 n=10+10) EncodeTwainCompress1e4-8 426µs ± 1% 428µs ± 2% ~ (p=0.549 n=9+10) EncodeTwainCompress1e5-8 6.80ms ± 1% 6.85ms ± 3% ~ (p=0.156 n=9+10) EncodeTwainCompress1e6-8 74.6ms ± 3% 73.8ms ± 2% ~ (p=0.280 n=10+10) name old speed new speed delta EncodeDigitsHuffman1e4-8 200MB/s ± 1% 203MB/s ± 1% +1.23% (p=0.000 n=10+10) EncodeDigitsHuffman1e5-8 210MB/s ± 1% 212MB/s ± 3% ~ (p=0.356 n=10+9) EncodeDigitsHuffman1e6-8 208MB/s ± 2% 210MB/s ± 2% ~ (p=0.243 n=10+9) EncodeDigitsSpeed1e4-8 32.8MB/s ± 3% 33.4MB/s ± 1% +1.88% (p=0.005 n=10+10) EncodeDigitsSpeed1e5-8 27.2MB/s ± 2% 27.9MB/s ± 1% +2.60% (p=0.000 n=10+8) EncodeDigitsSpeed1e6-8 26.1MB/s ± 2% 27.0MB/s ± 1% +3.56% (p=0.000 n=9+9) EncodeDigitsDefault1e4-8 27.7MB/s ± 2% 28.4MB/s ± 1% +2.24% (p=0.000 n=10+10) EncodeDigitsDefault1e5-8 19.1MB/s ± 2% 19.3MB/s ± 2% ~ (p=0.101 n=10+10) EncodeDigitsDefault1e6-8 17.7MB/s ± 3% 18.1MB/s ± 1% +2.46% (p=0.001 n=10+10) EncodeDigitsCompress1e4-8 27.6MB/s ± 2% 27.9MB/s ± 2% ~ (p=0.119 n=10+10) EncodeDigitsCompress1e5-8 19.0MB/s ± 3% 19.2MB/s ± 1% ~ (p=0.085 n=10+10) EncodeDigitsCompress1e6-8 17.9MB/s ± 4% 18.1MB/s ± 3% ~ (p=0.110 n=10+10) EncodeTwainHuffman1e4-8 141MB/s ± 3% 148MB/s ± 2% +4.79% (p=0.000 n=10+10) EncodeTwainHuffman1e5-8 180MB/s ± 2% 188MB/s ± 1% +4.38% (p=0.000 n=10+10) EncodeTwainHuffman1e6-8 181MB/s ± 3% 189MB/s ± 1% +4.54% (p=0.000 n=10+9) EncodeTwainSpeed1e4-8 34.0MB/s ± 3% 34.1MB/s ± 1% ~ (p=0.948 n=10+8) EncodeTwainSpeed1e5-8 38.7MB/s ± 2% 39.0MB/s ± 1% ~ (p=0.353 n=10+10) EncodeTwainSpeed1e6-8 39.1MB/s ± 1% 40.1MB/s ± 1% +2.68% (p=0.000 n=9+10) EncodeTwainDefault1e4-8 23.9MB/s ± 2% 24.0MB/s ± 1% ~ (p=0.734 n=10+9) EncodeTwainDefault1e5-8 16.0MB/s ± 4% 16.2MB/s ± 1% ~ (p=0.210 n=10+10) EncodeTwainDefault1e6-8 15.1MB/s ± 2% 15.2MB/s ± 1% ~ (p=0.515 n=10+10) EncodeTwainCompress1e4-8 23.5MB/s ± 1% 23.4MB/s ± 2% ~ (p=0.536 n=9+10) EncodeTwainCompress1e5-8 14.7MB/s ± 1% 14.6MB/s ± 3% ~ (p=0.138 n=9+10) EncodeTwainCompress1e6-8 13.4MB/s ± 3% 13.5MB/s ± 2% ~ (p=0.239 n=10+10) This improves "random input" to the dynamic writer, which is why the test data is updated. The output size goes from 1051 to 1005 bytes. Change-Id: I3ee11d2d2511b277d2dd16734aeea07c98bca450 Reviewed-on: https://go-review.googlesource.com/21757 Reviewed-by: Joe Tsai <joetsai@digital-static.net> Run-TryBot: Joe Tsai <joetsai@digital-static.net> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Nigel Tao <nigeltao@golang.org>
* all: remove unnecessary type conversionsMatthew Dempsky2016-04-154-7/+7
| | | | | | | | | | | | | cmd and runtime were handled separately, and I'm intentionally skipped syscall. This is the rest of the standard library. CL generated mechanically with github.com/mdempsky/unconvert. Change-Id: I9e0eff886974dedc37adb93f602064b83e469122 Reviewed-on: https://go-review.googlesource.com/22104 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: fix a fmt.Fprintf style nit in a test.Nigel Tao2016-04-151-1/+1
| | | | | | | | | | It's not a big deal (the for loop drops from 130-ish to 120-ish milliseconds for me) but it's not a big change either. Change-Id: I161a49caab5cae5a2b87866ed1dfb93627be8013 Reviewed-on: https://go-review.googlesource.com/22110 Reviewed-by: Klaus Post <klauspost@gmail.com> Reviewed-by: Nigel Tao <nigeltao@golang.org>
* compress/gzip: fix Reader to properly check FHCRCJoe Tsai2016-04-143-82/+107
| | | | | | | | | | | | | | | | | | | | | RFC 1952, section 3.2.3 says: >>> If FHCRC is set, a CRC16 for the gzip header is present, immediately before the compressed data. The CRC16 consists of the two least significant bytes of the CRC32 for all bytes of the gzip header up to and not including the CRC16. <<< Thus, instead of computing the CRC only over the first 10 bytes of the header, we compute it over the whole header (minus CRC16). Fixes #15070 Change-Id: I55703fd30b535b12abeb5e3962d4da0a86ed615a Reviewed-on: https://go-review.googlesource.com/21466 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: improve short writer error testKlaus Post2016-04-101-3/+7
| | | | | | | | | | | | | | | | | This improves the short version of the writer test. First of all, it has a much quicker setup. Previously that could take up towards 0.5 second. Secondly, it will test all compression levels in short mode as well. Execution time is 1.7s/0.03s for normal/short mode. Change-Id: I275a21f712daff6f7125cc6a493415e86439cb19 Reviewed-on: https://go-review.googlesource.com/21800 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: eliminate most common bounds checksKlaus Post2016-04-103-48/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This uses the SSA compiler to eliminate various unneeded bounds checks in loops and various lookups. This fixes the low hanging fruit, without any major code changes. name old time/op new time/op delta EncodeDigitsHuffman1e4-8 49.9µs ± 1% 48.1µs ± 1% -3.74% (p=0.000 n=10+9) EncodeDigitsHuffman1e5-8 476µs ± 1% 458µs ± 1% -3.58% (p=0.000 n=10+10) EncodeDigitsHuffman1e6-8 4.80ms ± 2% 4.56ms ± 1% -5.07% (p=0.000 n=10+9) EncodeDigitsSpeed1e4-8 305µs ± 3% 290µs ± 2% -5.03% (p=0.000 n=10+9) EncodeDigitsSpeed1e5-8 3.67ms ± 2% 3.49ms ± 2% -4.78% (p=0.000 n=9+10) EncodeDigitsSpeed1e6-8 38.3ms ± 2% 35.8ms ± 1% -6.58% (p=0.000 n=9+10) EncodeDigitsDefault1e4-8 361µs ± 2% 346µs ± 3% -4.12% (p=0.000 n=10+9) EncodeDigitsDefault1e5-8 5.24ms ± 2% 4.96ms ± 3% -5.38% (p=0.000 n=10+10) EncodeDigitsDefault1e6-8 56.5ms ± 3% 52.2ms ± 2% -7.68% (p=0.000 n=10+10) EncodeDigitsCompress1e4-8 362µs ± 2% 343µs ± 1% -5.20% (p=0.000 n=10+9) EncodeDigitsCompress1e5-8 5.26ms ± 3% 4.98ms ± 2% -5.48% (p=0.000 n=10+10) EncodeDigitsCompress1e6-8 56.0ms ± 4% 52.1ms ± 1% -7.01% (p=0.000 n=10+10) EncodeTwainHuffman1e4-8 70.9µs ± 3% 64.7µs ± 1% -8.68% (p=0.000 n=10+9) EncodeTwainHuffman1e5-8 556µs ± 2% 524µs ± 2% -5.84% (p=0.000 n=10+10) EncodeTwainHuffman1e6-8 5.54ms ± 3% 5.22ms ± 2% -5.70% (p=0.000 n=10+10) EncodeTwainSpeed1e4-8 294µs ± 3% 284µs ± 1% -3.71% (p=0.000 n=10+10) EncodeTwainSpeed1e5-8 2.59ms ± 2% 2.48ms ± 1% -4.14% (p=0.000 n=10+9) EncodeTwainSpeed1e6-8 25.6ms ± 1% 24.3ms ± 1% -5.28% (p=0.000 n=9+10) EncodeTwainDefault1e4-8 419µs ± 2% 396µs ± 1% -5.59% (p=0.000 n=10+9) EncodeTwainDefault1e5-8 6.23ms ± 4% 5.75ms ± 1% -7.83% (p=0.000 n=10+9) EncodeTwainDefault1e6-8 66.2ms ± 2% 61.4ms ± 1% -7.22% (p=0.000 n=10+10) EncodeTwainCompress1e4-8 426µs ± 1% 405µs ± 1% -4.97% (p=0.000 n=9+10) EncodeTwainCompress1e5-8 6.80ms ± 1% 6.32ms ± 1% -6.97% (p=0.000 n=9+10) EncodeTwainCompress1e6-8 74.6ms ± 3% 68.7ms ± 1% -7.90% (p=0.000 n=10+9) name old speed new speed delta EncodeDigitsHuffman1e4-8 200MB/s ± 1% 208MB/s ± 1% +3.88% (p=0.000 n=10+9) EncodeDigitsHuffman1e5-8 210MB/s ± 1% 218MB/s ± 1% +3.71% (p=0.000 n=10+10) EncodeDigitsHuffman1e6-8 208MB/s ± 2% 219MB/s ± 1% +5.32% (p=0.000 n=10+9) EncodeDigitsSpeed1e4-8 32.8MB/s ± 3% 34.5MB/s ± 2% +5.29% (p=0.000 n=10+9) EncodeDigitsSpeed1e5-8 27.2MB/s ± 2% 28.6MB/s ± 2% +5.29% (p=0.000 n=10+10) EncodeDigitsSpeed1e6-8 26.1MB/s ± 2% 27.9MB/s ± 1% +7.02% (p=0.000 n=9+10) EncodeDigitsDefault1e4-8 27.7MB/s ± 2% 28.9MB/s ± 3% +4.30% (p=0.000 n=10+9) EncodeDigitsDefault1e5-8 19.1MB/s ± 2% 20.2MB/s ± 3% +5.69% (p=0.000 n=10+10) EncodeDigitsDefault1e6-8 17.7MB/s ± 3% 19.2MB/s ± 2% +8.31% (p=0.000 n=10+10) EncodeDigitsCompress1e4-8 27.6MB/s ± 2% 29.1MB/s ± 1% +5.47% (p=0.000 n=10+9) EncodeDigitsCompress1e5-8 19.0MB/s ± 3% 20.1MB/s ± 2% +5.78% (p=0.000 n=10+10) EncodeDigitsCompress1e6-8 17.9MB/s ± 4% 19.2MB/s ± 1% +7.50% (p=0.000 n=10+10) EncodeTwainHuffman1e4-8 141MB/s ± 3% 154MB/s ± 1% +9.46% (p=0.000 n=10+9) EncodeTwainHuffman1e5-8 180MB/s ± 2% 191MB/s ± 2% +6.19% (p=0.000 n=10+10) EncodeTwainHuffman1e6-8 181MB/s ± 3% 192MB/s ± 2% +6.02% (p=0.000 n=10+10) EncodeTwainSpeed1e4-8 34.0MB/s ± 3% 35.3MB/s ± 1% +3.84% (p=0.000 n=10+10) EncodeTwainSpeed1e5-8 38.7MB/s ± 2% 40.3MB/s ± 1% +4.30% (p=0.000 n=10+9) EncodeTwainSpeed1e6-8 39.1MB/s ± 1% 41.2MB/s ± 1% +5.57% (p=0.000 n=9+10) EncodeTwainDefault1e4-8 23.9MB/s ± 2% 25.3MB/s ± 1% +5.91% (p=0.000 n=10+9) EncodeTwainDefault1e5-8 16.0MB/s ± 4% 17.4MB/s ± 1% +8.47% (p=0.000 n=10+9) EncodeTwainDefault1e6-8 15.1MB/s ± 2% 16.3MB/s ± 1% +7.76% (p=0.000 n=10+10) EncodeTwainCompress1e4-8 23.5MB/s ± 1% 24.7MB/s ± 1% +5.24% (p=0.000 n=9+10) EncodeTwainCompress1e5-8 14.7MB/s ± 1% 15.8MB/s ± 1% +7.50% (p=0.000 n=9+10) EncodeTwainCompress1e6-8 13.4MB/s ± 3% 14.6MB/s ± 1% +8.57% (p=0.000 n=10+9) Change-Id: I5c7e84c2f9ea4d38a2115995705eebb93387e22f Reviewed-on: https://go-review.googlesource.com/21759 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/gzip: fix Reader.ResetJoe Tsai2016-04-022-45/+43
| | | | | | | | | | | | | | Rather than specifying every field that should be cleared in Reset, it is better to just zero the entire struct and only preserve or set the fields that we actually care about. This ensures that the Header field is reset for the next use. Fixes #15077 Change-Id: I41832e506d2d64c62b700aa1986e7de24a577511 Reviewed-on: https://go-review.googlesource.com/21465 Run-TryBot: Joe Tsai <joetsai@digital-static.net> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* compress/gzip: cleanup gzip packageJoe Tsai2016-04-023-50/+32
| | | | | | | | | | | | | | | | | | | Changes made: * Reader.flg is not used anywhere else other than readHeader and does not need to be stored. * Store Reader.digest and Writer.digest as uint32s rather than as a hash.Hash32 and use the crc32.Update function instead. This simplifies initialization logic since the zero value of uint32 is the initial CRC-32 value. There are no performance detriments to doing this since the hash.Hash32 returned by crc32 simply calls crc32.Update as well. * s/[0:/[:/ Consistently use shorter notation for slicing. * s/RFC1952/RFC 1952/ Consistently use RFC notation. Change-Id: I55416a19f4836cbed943adaa3f672538ea5d166d Reviewed-on: https://go-review.googlesource.com/21429 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Joe Tsai <joetsai@digital-static.net> TryBot-Result: Gobot Gobot <gobot@golang.org>
* all: use bytes.Equal, bytes.Contains and strings.ContainsDominik Honnef2016-04-011-1/+1
| | | | | | | Change-Id: Iba82a5bd3846f7ab038cc10ec72ff6bcd2c0b484 Reviewed-on: https://go-review.googlesource.com/21377 Run-TryBot: Dave Cheney <dave@cheney.net> Reviewed-by: Dave Cheney <dave@cheney.net>
* compress/flate: make Reader.Read return io.EOF eagerlyJoe Tsai2016-03-312-12/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rather than checking the block final bit on the next invocation of nextBlock, we check it at the termination of the current block. This ensures that we return (n, io.EOF) instead of (0, io.EOF) more frequently for most streams. However, there are certain situations where an eager io.EOF is not done: 1) We previously returned from Read because the write buffer of the internal dictionary was full, and it just so happens that there is no more data remaining in the stream. 2) There exists a [non-final, empty, raw block] after all blocks that actually contain uncompressed data. We cannot return io.EOF eagerly here since it would break flushing semantics. Both situations happen infrequently, but it is still important to note that this change does *not* guarantee that flate will *always* return (n, io.EOF). Furthermore, this CL makes no changes to the pattern of ReadByte calls to the underlying io.ByteReader. Below is the motivation for this change, pulling the text from @bradfitz's CL/21290: net/http and other things work better when io.Reader implementations return (n, io.EOF) at the end, instead of (n, nil) followed by (0, io.EOF). Both are legal, but the standard library has been moving towards n+io.EOF. An investigation of net/http connection re-use in https://github.com/google/go-github/pull/317 revealed that with gzip compression + http/1.1 chunking, the net/http package was not automatically reusing the underlying TCP connections when the final EOF bytes were already read off the wire. The net/http package only reuses the connection if the underlying Readers (many of them nested in this case) all eagerly return io.EOF. Previous related CLs: https://golang.org/cl/76400046 - tls.Reader https://golang.org/cl/58240043 - http chunked reader In addition to net/http, this behavior also helps things like ioutil.ReadAll (see comments about performance improvements in https://codereview.appspot.com/49570044) Updates #14867 Updates google/go-github#317 Change-Id: I637c45552efb561d34b13ed918b73c660f668378 Reviewed-on: https://go-review.googlesource.com/21302 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* compress/gzip: fix error handling in ReadJoe Tsai2016-03-301-17/+17
| | | | | | | | | | | | | | The Read logic should not assume that only (0, io.EOF) is returned instead of (n, io.EOF) where n is positive. The fix done here is very similar to the fix to compress/zlib in CL/20292. Change-Id: Icb76258cdcf8cfa386a60bab330fefde46fc071d Reviewed-on: https://go-review.googlesource.com/21308 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: add pure huffman deflaterKlaus Post2016-03-294-19/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a "HuffmanOnly" compression level, where the input is only entropy encoded. The output is fully inflate compatible. Typical compression is reduction is about 50% of typical level 1 compression, however the compression time is very stable, and does not vary as much as nearly as much level 1 compression (or Snappy). This mode is useful for: * HTTP compression in a CPU limited environment. * Entropy encoding Snappy compressed data, for archiving, etc. * Compression where compression time needs to be predictable. * Fast network transfer. Snappy "usually" performs inbetween this and level 1 compression-wise, but at the same speed as "Huffman", so this is not a replacement, but a good supplement for Snappy, since it usually can compress Snappy output further. This is implemented as level -2, since this would be too much of a compression reduction to replace level 1. >go test -bench=Encode -cpu=1 BenchmarkEncodeDigitsHuffman1e4 30000 52334 ns/op 191.08 MB/s BenchmarkEncodeDigitsHuffman1e5 3000 518343 ns/op 192.92 MB/s BenchmarkEncodeDigitsHuffman1e6 300 5356884 ns/op 186.68 MB/s BenchmarkEncodeDigitsSpeed1e4 5000 324214 ns/op 30.84 MB/s BenchmarkEncodeDigitsSpeed1e5 500 3952614 ns/op 25.30 MB/s BenchmarkEncodeDigitsSpeed1e6 30 40760350 ns/op 24.53 MB/s BenchmarkEncodeDigitsDefault1e4 5000 387056 ns/op 25.84 MB/s BenchmarkEncodeDigitsDefault1e5 300 5950614 ns/op 16.80 MB/s BenchmarkEncodeDigitsDefault1e6 20 63842195 ns/op 15.66 MB/s BenchmarkEncodeDigitsCompress1e4 5000 391859 ns/op 25.52 MB/s BenchmarkEncodeDigitsCompress1e5 300 5707112 ns/op 17.52 MB/s BenchmarkEncodeDigitsCompress1e6 20 59839465 ns/op 16.71 MB/s BenchmarkEncodeTwainHuffman1e4 20000 73498 ns/op 136.06 MB/s BenchmarkEncodeTwainHuffman1e5 2000 595892 ns/op 167.82 MB/s BenchmarkEncodeTwainHuffman1e6 200 6059016 ns/op 165.04 MB/s BenchmarkEncodeTwainSpeed1e4 5000 321212 ns/op 31.13 MB/s BenchmarkEncodeTwainSpeed1e5 500 2823873 ns/op 35.41 MB/s BenchmarkEncodeTwainSpeed1e6 50 27237864 ns/op 36.71 MB/s BenchmarkEncodeTwainDefault1e4 3000 454634 ns/op 22.00 MB/s BenchmarkEncodeTwainDefault1e5 200 6859537 ns/op 14.58 MB/s BenchmarkEncodeTwainDefault1e6 20 71547405 ns/op 13.98 MB/s BenchmarkEncodeTwainCompress1e4 3000 462307 ns/op 21.63 MB/s BenchmarkEncodeTwainCompress1e5 200 7534992 ns/op 13.27 MB/s BenchmarkEncodeTwainCompress1e6 20 80353365 ns/op 12.45 MB/s PASS ok compress/flate 55.333s Change-Id: I8e12ad13220e50d4cf7ddba6f292333efad61b0c Reviewed-on: https://go-review.googlesource.com/20982 Reviewed-by: Joe Tsai <joetsai@digital-static.net> Reviewed-by: Nigel Tao <nigeltao@golang.org>
* compress/gzip: clean up zero-mtimes testCaleb Spare2016-03-251-1/+5
| | | | | | | | | | - Fix a typo. - Skip this test on -short on non-builders. Change-Id: Id102eceb59451694bf92b618e02ccee6603b6852 Reviewed-on: https://go-review.googlesource.com/21113 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
* compress/gzip: skip mtime test when GOROOT doesn't existElias Naur2016-03-251-0/+3
| | | | | | | | Fixes the iOS builders Change-Id: I5097ca19048381ffb5a4c5ea038b7c4aa18ee4b7 Reviewed-on: https://go-review.googlesource.com/21132 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* all: delete dead non-test codeDominik Honnef2016-03-252-4/+0
| | | | | | | | | | | | | | | | | | | | | | | This change removes a lot of dead code. Some of the code has never been used, not even when it was first commited. The rest shouldn't have survived refactors. This change doesn't remove unused routines helpful for debugging, nor does it remove code that's used in commented out blocks of code that are only unused temporarily. Furthermore, unused constants weren't removed when they were part of a set of constants from specifications. One noteworthy omission from this CL are about 1000 lines of unused code in cmd/fix, 700 lines of which are the typechecker, which hasn't been used ever since the pre-Go 1 fixes have been removed. I wasn't sure if this code should stick around for future uses of cmd/fix or be culled as well. Change-Id: Ib714bc7e487edc11ad23ba1c3222d1fd02e4a549 Reviewed-on: https://go-review.googlesource.com/20926 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* all: zero mtimes in testdata gz filesCaleb Spare2016-03-252-0/+52
| | | | | | | | | | Fixes #14937. Change-Id: Iea11a32230d44d5a43f8aec812d25f13bce85895 Reviewed-on: https://go-review.googlesource.com/21038 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* compress/flate: rework matching algorithmKlaus Post2016-03-232-90/+210
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes how matching is done in deflate algorithm. The major change is that we do not look for matches that are only 3 bytes in length, matches must be 4 bytes at least. Contrary to what you would expect this actually improves the compresion ratio, since 3 literal bytes will often be shorter than a match after huffman encoding. This varies a bit by source, but is most often the case when the source is "easy" to compress. Second of all, a "stronger" hash is used. The hash is similar to the hashing function used by Snappy. Overall, the speed impact is biggest on higher compression levels. I intend to replace the "speed" compression level, which can be seen in CL 21021. The built-in benchmark using "digits" is slower at level 1. I see this as an exception, since "digits" is a special type of data, where you have low entropy (numbers 0->9), but no significant matches. Again, CL 20021 fixes that case. NewWriterDict is also made considerably faster, by not running data through the entire encoder. This is not reflected by the benchmark. Overall, the speed impact is biggest on higher compression levels. I intend to replace the "speed" compression level. COMPARED to tip/master: name old time/op new time/op delta EncodeDigitsSpeed1e4-4 401µs ± 1% 345µs ± 2% -13.95% EncodeDigitsSpeed1e5-4 3.19ms ± 1% 4.27ms ± 3% +33.96% EncodeDigitsSpeed1e6-4 27.7ms ± 4% 43.8ms ± 3% +58.00% EncodeDigitsDefault1e4-4 641µs ± 0% 403µs ± 1% -37.15% EncodeDigitsDefault1e5-4 13.8ms ± 1% 6.4ms ± 3% -53.73% EncodeDigitsDefault1e6-4 162ms ± 1% 64ms ± 2% -60.51% EncodeDigitsCompress1e4-4 627µs ± 1% 405µs ± 2% -35.45% EncodeDigitsCompress1e5-4 13.9ms ± 0% 6.3ms ± 2% -54.46% EncodeDigitsCompress1e6-4 159ms ± 1% 64ms ± 0% -59.91% EncodeTwainSpeed1e4-4 433µs ± 4% 331µs ± 1% -23.53% EncodeTwainSpeed1e5-4 2.82ms ± 1% 3.08ms ± 0% +9.10% EncodeTwainSpeed1e6-4 28.1ms ± 2% 28.8ms ± 0% +2.82% EncodeTwainDefault1e4-4 695µs ± 4% 474µs ± 1% -31.78% EncodeTwainDefault1e5-4 11.8ms ± 0% 7.4ms ± 0% -37.31% EncodeTwainDefault1e6-4 128ms ± 0% 75ms ± 0% -40.93% EncodeTwainCompress1e4-4 719µs ± 3% 480µs ± 0% -33.27% EncodeTwainCompress1e5-4 15.0ms ± 3% 8.2ms ± 2% -45.55% EncodeTwainCompress1e6-4 170ms ± 0% 85ms ± 1% -49.99% name old speed new speed delta EncodeDigitsSpeed1e4-4 25.0MB/s ± 1% 29.0MB/s ± 2% +16.24% EncodeDigitsSpeed1e5-4 31.4MB/s ± 1% 23.4MB/s ± 3% -25.34% EncodeDigitsSpeed1e6-4 36.1MB/s ± 4% 22.8MB/s ± 3% -36.74% EncodeDigitsDefault1e4-4 15.6MB/s ± 0% 24.8MB/s ± 1% +59.11% EncodeDigitsDefault1e5-4 7.27MB/s ± 1% 15.72MB/s ± 3% +116.23% EncodeDigitsDefault1e6-4 6.16MB/s ± 0% 15.60MB/s ± 2% +153.25% EncodeDigitsCompress1e4-4 15.9MB/s ± 1% 24.7MB/s ± 2% +54.97% EncodeDigitsCompress1e5-4 7.19MB/s ± 0% 15.78MB/s ± 2% +119.62% EncodeDigitsCompress1e6-4 6.27MB/s ± 1% 15.65MB/s ± 0% +149.52% EncodeTwainSpeed1e4-4 23.1MB/s ± 4% 30.2MB/s ± 1% +30.68% EncodeTwainSpeed1e5-4 35.4MB/s ± 1% 32.5MB/s ± 0% -8.34% EncodeTwainSpeed1e6-4 35.6MB/s ± 2% 34.7MB/s ± 0% -2.77% EncodeTwainDefault1e4-4 14.4MB/s ± 4% 21.1MB/s ± 1% +46.48% EncodeTwainDefault1e5-4 8.49MB/s ± 0% 13.55MB/s ± 0% +59.50% EncodeTwainDefault1e6-4 7.83MB/s ± 0% 13.25MB/s ± 0% +69.19% EncodeTwainCompress1e4-4 13.9MB/s ± 3% 20.8MB/s ± 0% +49.83% EncodeTwainCompress1e5-4 6.65MB/s ± 3% 12.20MB/s ± 2% +83.51% EncodeTwainCompress1e6-4 5.88MB/s ± 0% 11.76MB/s ± 1% +100.06% Change-Id: I724e33c1dd3e3a6a1b0a68e094baa959352baf32 Reviewed-on: https://go-review.googlesource.com/20929 Run-TryBot: Nigel Tao <nigeltao@golang.org> Reviewed-by: Nigel Tao <nigeltao@golang.org>
* compress/flate: optimize huffman bit encoderKlaus Post2016-03-1155-180/+775
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Part 1 of optimizing the deflater. This optimizes the bitwriter by: * Removing allocations. * Storing compound values for bit codes instead of 2 separate tables. * Accumulate 48 bits between writes instead of 24. * Inline bit flushing. This also contains code that will be used in later CL's (writeBlockDynamic, writeBlockHuff). Tests for Huffman bit writer encoding regressions has been added. name old speed new speed delta EncodeDigitsSpeed1e4-4 19.3MB/s ± 1% 21.6MB/s ± 1% +11.77% EncodeDigitsSpeed1e5-4 25.0MB/s ± 6% 30.7MB/s ± 1% +22.70% EncodeDigitsSpeed1e6-4 28.2MB/s ± 1% 32.3MB/s ± 1% +14.64% EncodeDigitsDefault1e4-4 13.3MB/s ± 0% 14.2MB/s ± 1% +7.07% EncodeDigitsDefault1e5-4 6.43MB/s ± 1% 6.64MB/s ± 1% +3.27% EncodeDigitsDefault1e6-4 5.81MB/s ± 0% 5.85MB/s ± 1% +0.69% EncodeDigitsCompress1e4-4 13.2MB/s ± 0% 14.4MB/s ± 0% +9.10% EncodeDigitsCompress1e5-4 6.40MB/s ± 1% 6.61MB/s ± 0% +3.20% EncodeDigitsCompress1e6-4 5.80MB/s ± 1% 5.90MB/s ± 1% +1.64% EncodeTwainSpeed1e4-4 18.4MB/s ± 1% 20.7MB/s ± 1% +12.72% EncodeTwainSpeed1e5-4 27.7MB/s ± 1% 31.0MB/s ± 1% +11.78% EncodeTwainSpeed1e6-4 29.1MB/s ± 0% 32.9MB/s ± 2% +13.25% EncodeTwainDefault1e4-4 12.4MB/s ± 0% 13.1MB/s ± 1% +5.88% EncodeTwainDefault1e5-4 7.52MB/s ± 1% 7.83MB/s ± 0% +4.19% EncodeTwainDefault1e6-4 7.08MB/s ± 1% 7.26MB/s ± 0% +2.54% EncodeTwainCompress1e4-4 12.0MB/s ± 1% 12.8MB/s ± 1% +6.70% EncodeTwainCompress1e5-4 5.96MB/s ± 1% 6.16MB/s ± 0% +3.27% EncodeTwainCompress1e6-4 5.37MB/s ± 0% 5.39MB/s ± 1% +0.47% >Allocations: benchmark old allocs new allocs delta BenchmarkEncodeDigitsSpeed1e4-4 50 0 -100.00% BenchmarkEncodeDigitsSpeed1e5-4 110 0 -100.00% BenchmarkEncodeDigitsSpeed1e6-4 1032 0 -100.00% BenchmarkEncodeDigitsDefault1e4-4 56 0 -100.00% BenchmarkEncodeDigitsDefault1e5-4 120 0 -100.00% BenchmarkEncodeDigitsDefault1e6-4 966 0 -100.00% BenchmarkEncodeDigitsCompress1e4-4 56 0 -100.00% BenchmarkEncodeDigitsCompress1e5-4 120 0 -100.00% BenchmarkEncodeDigitsCompress1e6-4 966 0 -100.00% BenchmarkEncodeTwainSpeed1e4-4 58 0 -100.00% BenchmarkEncodeTwainSpeed1e5-4 132 0 -100.00% BenchmarkEncodeTwainSpeed1e6-4 1082 0 -100.00% BenchmarkEncodeTwainDefault1e4-4 52 0 -100.00% BenchmarkEncodeTwainDefault1e5-4 126 0 -100.00% BenchmarkEncodeTwainDefault1e6-4 886 0 -100.00% BenchmarkEncodeTwainCompress1e4-4 52 0 -100.00% BenchmarkEncodeTwainCompress1e5-4 120 0 -100.00% BenchmarkEncodeTwainCompress1e6-4 880 0 -100.00% benchmark old bytes new bytes delta BenchmarkEncodeDigitsSpeed1e4-4 4288 2 -99.95% BenchmarkEncodeDigitsSpeed1e5-4 8896 15 -99.83% BenchmarkEncodeDigitsSpeed1e6-4 84098 153 -99.82% BenchmarkEncodeDigitsDefault1e4-4 4480 3 -99.93% BenchmarkEncodeDigitsDefault1e5-4 9216 76 -99.18% BenchmarkEncodeDigitsDefault1e6-4 73920 768 -98.96% BenchmarkEncodeDigitsCompress1e4-4 4480 3 -99.93% BenchmarkEncodeDigitsCompress1e5-4 9216 76 -99.18% BenchmarkEncodeDigitsCompress1e6-4 73920 768 -98.96% BenchmarkEncodeTwainSpeed1e4-4 4544 2 -99.96% BenchmarkEncodeTwainSpeed1e5-4 9600 15 -99.84% BenchmarkEncodeTwainSpeed1e6-4 77633 153 -99.80% BenchmarkEncodeTwainDefault1e4-4 4352 3 -99.93% BenchmarkEncodeTwainDefault1e5-4 9408 76 -99.19% BenchmarkEncodeTwainDefault1e6-4 65984 768 -98.84% BenchmarkEncodeTwainCompress1e4-4 4352 3 -99.93% BenchmarkEncodeTwainCompress1e5-4 9216 76 -99.18% BenchmarkEncodeTwainCompress1e6-4 65792 768 -98.83% Updates #14258 Change-Id: Ibaa97b9619743ad623094727228eb2ada1ec7f1f Reviewed-on: https://go-review.googlesource.com/19336 Reviewed-by: Nigel Tao <nigeltao@golang.org> Reviewed-by: Joe Tsai <joetsai@digital-static.net> Run-TryBot: Joe Tsai <joetsai@digital-static.net> TryBot-Result: Gobot Gobot <gobot@golang.org>
View Lorry Controller Status