delta/go-git.git/src/encoding, branch dev.typeparams github.com: golang/go encoding/csv: add FieldPos method 2021-04-29T17:47:48+00:00 Roger Peppe rogpeppe@gmail.com 2021-02-11T13:54:45+00:00 6d95e5a4ff9f2ffa12ca225974d0ce45fd2504d4 This enables a consumer of a CSV to find out the position of a CSV field without implementing an intermediate buffer. This is useful to produce good higher level error messages when the CSV syntax is OK but the field values don't match expectations. This also changes the existing semantics of the `ParseError.Column` field to bring it in line with precedent elsewhere in the Go standard library (notably go/token.Position) - the column is now 1-based and indicates a byte count rather than a rune count, and the error position reporting at the end of a last line without a newline is now fixed. This change has some impact on performance: ``` name old time/op new time/op delta Read-8 2.14µs ± 0% 2.15µs ± 0% ~ (p=0.056 n=5+5) ReadWithFieldsPerRecord-8 2.15µs ± 2% 2.15µs ± 1% ~ (p=0.151 n=5+5) ReadWithoutFieldsPerRecord-8 2.15µs ± 0% 2.15µs ± 0% +0.37% (p=0.024 n=5+5) ReadLargeFields-8 3.55µs ± 2% 3.59µs ± 0% ~ (p=0.206 n=5+5) ReadReuseRecord-8 1.18µs ± 1% 1.22µs ± 1% +2.93% (p=0.008 n=5+5) ReadReuseRecordWithFieldsPerRecord-8 1.18µs ± 0% 1.21µs ± 0% +2.54% (p=0.008 n=5+5) ReadReuseRecordWithoutFieldsPerRecord-8 1.18µs ± 0% 1.22µs ± 1% +3.66% (p=0.008 n=5+5) ReadReuseRecordLargeFields-8 2.53µs ± 1% 2.57µs ± 1% +1.70% (p=0.008 n=5+5) Write-8 1.02µs ± 1% 1.01µs ± 0% -1.18% (p=0.016 n=5+4) ``` Fixes #44221. Change-Id: Id37c50fc396024eef406c5bad45380ecd414f5ea Reviewed-on: https://go-review.googlesource.com/c/go/+/291290 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Trust: Paul Jolly <paul@myitcv.org.uk> 
This enables a consumer of a CSV to find out the position
of a CSV field without implementing an intermediate buffer.
This is useful to produce good higher level error messages when
the CSV syntax is OK but the field values don't match expectations.

This also changes the existing semantics of the `ParseError.Column`
field to bring it in line with precedent elsewhere in the Go
standard library (notably go/token.Position) - the column is
now 1-based and indicates a byte count rather than a rune count,
and the error position reporting at the end of a last line without
a newline is now fixed.

This change has some impact on performance:

```
name                                     old time/op    new time/op    delta
Read-8                                     2.14µs ± 0%    2.15µs ± 0%    ~     (p=0.056 n=5+5)
ReadWithFieldsPerRecord-8                  2.15µs ± 2%    2.15µs ± 1%    ~     (p=0.151 n=5+5)
ReadWithoutFieldsPerRecord-8               2.15µs ± 0%    2.15µs ± 0%  +0.37%  (p=0.024 n=5+5)
ReadLargeFields-8                          3.55µs ± 2%    3.59µs ± 0%    ~     (p=0.206 n=5+5)
ReadReuseRecord-8                          1.18µs ± 1%    1.22µs ± 1%  +2.93%  (p=0.008 n=5+5)
ReadReuseRecordWithFieldsPerRecord-8       1.18µs ± 0%    1.21µs ± 0%  +2.54%  (p=0.008 n=5+5)
ReadReuseRecordWithoutFieldsPerRecord-8    1.18µs ± 0%    1.22µs ± 1%  +3.66%  (p=0.008 n=5+5)
ReadReuseRecordLargeFields-8               2.53µs ± 1%    2.57µs ± 1%  +1.70%  (p=0.008 n=5+5)
Write-8                                    1.02µs ± 1%    1.01µs ± 0%  -1.18%  (p=0.016 n=5+4)
```

Fixes #44221.

Change-Id: Id37c50fc396024eef406c5bad45380ecd414f5ea
Reviewed-on: https://go-review.googlesource.com/c/go/+/291290
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Paul Jolly <paul@myitcv.org.uk>
all: remove redundant spaces before . and , 2021-04-20T00:49:17+00:00 Yury Smolsky yury@smolsky.by 2021-04-17T09:12:28+00:00 4f5aec46039347c7abc02a10103f35f936f39c2b Change-Id: I6a4bd2544276d0638bddf07ebcf2ee636db30fea Reviewed-on: https://go-review.googlesource.com/c/go/+/311009 Run-TryBot: Yury Smolsky <yury@smolsky.by> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Trust: Emmanuel Odeke <emmanuel@orijtech.com> 
Change-Id: I6a4bd2544276d0638bddf07ebcf2ee636db30fea
Reviewed-on: https://go-review.googlesource.com/c/go/+/311009
Run-TryBot: Yury Smolsky <yury@smolsky.by>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
encoding/xml: replace comments inside directives with a space 2021-03-15T20:04:23+00:00 Filippo Valsorda filippo@golang.org 2020-10-26T23:21:30+00:00 a9cfd55e2b09735a25976d1b008a0a3c767494f8 A Directive (like <!ENTITY xxx []>) can't have other nodes nested inside it (in our data structure representation), so there is no way to preserve comments. The previous behavior was to just elide them, which however might change the semantic meaning of the surrounding markup. Instead, replace them with a space which hopefully has the same semantic effect of the comment. Directives are not actually a node type in the XML spec, which instead specifies each of them separately (<!ENTITY, <!DOCTYPE, etc.), each with its own grammar. The rules for where and when the comments are allowed are not straightforward, and can't be implemented without implementing custom logic for each of the directives. Simply preserving the comments in the body of the directive would be problematic, as there can be unmatched quotes inside the comment. Whether those quotes are considered meaningful semantically or not, other parsers might disagree and interpret the output differently. This issue was reported by Juho Nurminen of Mattermost as it leads to round-trip mismatches. See #43168. It's not being fixed in a security release because round-trip stability is not a currently supported security property of encoding/xml, and we don't believe these fixes would be sufficient to reliably guarantee it in the future. Fixes CVE-2020-29510 Updates #43168 Change-Id: Icd86c75beff3e1e0689543efebdad10ed5178ce3 Reviewed-on: https://go-review.googlesource.com/c/go/+/277893 Run-TryBot: Filippo Valsorda <filippo@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Filippo Valsorda <filippo@golang.org> Reviewed-by: Katie Hockman <katie@golang.org> 
A Directive (like <!ENTITY xxx []>) can't have other nodes nested inside
it (in our data structure representation), so there is no way to
preserve comments. The previous behavior was to just elide them, which
however might change the semantic meaning of the surrounding markup.
Instead, replace them with a space which hopefully has the same semantic
effect of the comment.

Directives are not actually a node type in the XML spec, which instead
specifies each of them separately (<!ENTITY, <!DOCTYPE, etc.), each with
its own grammar. The rules for where and when the comments are allowed
are not straightforward, and can't be implemented without implementing
custom logic for each of the directives.

Simply preserving the comments in the body of the directive would be
problematic, as there can be unmatched quotes inside the comment.
Whether those quotes are considered meaningful semantically or not,
other parsers might disagree and interpret the output differently.

This issue was reported by Juho Nurminen of Mattermost as it leads to
round-trip mismatches. See #43168. It's not being fixed in a security
release because round-trip stability is not a currently supported
security property of encoding/xml, and we don't believe these fixes
would be sufficient to reliably guarantee it in the future.

Fixes CVE-2020-29510
Updates #43168

Change-Id: Icd86c75beff3e1e0689543efebdad10ed5178ce3
Reviewed-on: https://go-review.googlesource.com/c/go/+/277893
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
encoding/xml: handle leading, trailing, or double colons in names 2021-03-15T20:04:12+00:00 Filippo Valsorda filippo@golang.org 2020-10-26T23:17:15+00:00 4d014e723165f28b34458edb4aa9136e0fb4c702 Before this change, <:name> would parse as <name>, which could cause issues in applications that rely on the parse-encode cycle to round-trip. Similarly, <x name:=""> would parse as expected but then have the attribute dropped when serializing because its name was empty. Finally, <a:b:c> would parse and get serialized incorrectly. All these values are invalid XML, but to minimize the impact of this change, we parse them whole into Name.Local. This issue was reported by Juho Nurminen of Mattermost as it leads to round-trip mismatches. See #43168. It's not being fixed in a security release because round-trip stability is not a currently supported security property of encoding/xml, and we don't believe these fixes would be sufficient to reliably guarantee it in the future. Fixes CVE-2020-29509 Fixes CVE-2020-29511 Updates #43168 Change-Id: I68321c4d867305046f664347192948a889af3c7f Reviewed-on: https://go-review.googlesource.com/c/go/+/277892 Run-TryBot: Filippo Valsorda <filippo@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Filippo Valsorda <filippo@golang.org> Reviewed-by: Katie Hockman <katie@golang.org> 
Before this change, <:name> would parse as <name>, which could cause
issues in applications that rely on the parse-encode cycle to
round-trip. Similarly, <x name:=""> would parse as expected but then
have the attribute dropped when serializing because its name was empty.
Finally, <a:b:c> would parse and get serialized incorrectly. All these
values are invalid XML, but to minimize the impact of this change, we
parse them whole into Name.Local.

This issue was reported by Juho Nurminen of Mattermost as it leads to
round-trip mismatches. See #43168. It's not being fixed in a security
release because round-trip stability is not a currently supported
security property of encoding/xml, and we don't believe these fixes
would be sufficient to reliably guarantee it in the future.

Fixes CVE-2020-29509
Fixes CVE-2020-29511
Updates #43168

Change-Id: I68321c4d867305046f664347192948a889af3c7f
Reviewed-on: https://go-review.googlesource.com/c/go/+/277892
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
encoding/json: fix package shadowing in MarshalIndent example 2021-03-14T19:58:14+00:00 Ariel Mashraki ariel@mashraki.co.il 2021-03-14T16:12:59+00:00 a8b59fe3cdaeeb40c87d55122a45a2e390e60d88 Prior to this CL, pasting the example from the website causes a compilation error for some programs because it was shadowing the "json" package. Change-Id: I39b68a66ca99468547f2027a7655cf1387b61e95 Reviewed-on: https://go-review.googlesource.com/c/go/+/301492 Reviewed-by: Joe Tsai <joetsai@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org> Trust: Joe Tsai <joetsai@google.com> Run-TryBot: Joe Tsai <joetsai@google.com> TryBot-Result: Go Bot <gobot@golang.org> 
Prior to this CL, pasting the example from the website causes a
compilation error for some programs because it was shadowing the
"json" package.

Change-Id: I39b68a66ca99468547f2027a7655cf1387b61e95
Reviewed-on: https://go-review.googlesource.com/c/go/+/301492
Reviewed-by: Joe Tsai <joetsai@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Joe Tsai <joetsai@google.com>
Run-TryBot: Joe Tsai <joetsai@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
encoding/gob: ensure "duplicate type received" decoder errors surface up 2021-03-13T11:52:17+00:00 Aman Karmani aman@tmm1.net 2021-03-02T19:52:34+00:00 b3235b75d109f06eec0d3603c606b2d8373b9d4c Previously re-using a decoder with a new stream resulted in a confusing "extra data in buffer" error message. Change-Id: Ia4c4c3a2d4b63c59e37e53faa61a500d5ff6e5f1 Reviewed-on: https://go-review.googlesource.com/c/go/+/297949 Reviewed-by: Rob Pike <r@golang.org> Run-TryBot: Rob Pike <r@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Emmanuel Odeke <emmanuel@orijtech.com> 
Previously re-using a decoder with a new stream resulted in a confusing
"extra data in buffer" error message.

Change-Id: Ia4c4c3a2d4b63c59e37e53faa61a500d5ff6e5f1
Reviewed-on: https://go-review.googlesource.com/c/go/+/297949
Reviewed-by: Rob Pike <r@golang.org>
Run-TryBot: Rob Pike <r@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
all: use HTML5 br tags 2021-03-13T03:38:42+00:00 John Bampton jbampton@gmail.com 2021-03-05T01:53:00+00:00 8e725f8452ac0ece548837a95d125bc67b9d8828 In HTML5 br tags don't need a closing slash Change-Id: Ic53c43faee08c5b1267daa9a02cc186b1c255ca1 GitHub-Last-Rev: 652208116944d01b23b8af8f1af485da5e916d32 GitHub-Pull-Request: golang/go#44283 Reviewed-on: https://go-review.googlesource.com/c/go/+/292370 Reviewed-by: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Emmanuel Odeke <emmanuel@orijtech.com> 
In HTML5 br tags don't need a closing slash

Change-Id: Ic53c43faee08c5b1267daa9a02cc186b1c255ca1
GitHub-Last-Rev: 652208116944d01b23b8af8f1af485da5e916d32
GitHub-Pull-Request: golang/go#44283
Reviewed-on: https://go-review.googlesource.com/c/go/+/292370
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
encoding/xml: prevent infinite loop while decoding 2021-03-10T18:19:03+00:00 Katie Hockman katie@golang.org 2021-03-01T14:54:00+00:00 d0b79e3513a29628f3599dc8860666b6eed75372 This change properly handles a TokenReader which returns an EOF in the middle of an open XML element. Thanks to Sam Whited for reporting this. Fixes CVE-2021-27918 Fixes #44913 Change-Id: Id02a3f3def4a1b415fa2d9a8e3b373eb6cb0f433 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/1004594 Reviewed-by: Russ Cox <rsc@google.com> Reviewed-by: Roland Shoemaker <bracewell@google.com> Reviewed-by: Filippo Valsorda <valsorda@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/300391 Trust: Katie Hockman <katie@golang.org> Run-TryBot: Katie Hockman <katie@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Filippo Valsorda <filippo@golang.org> 
This change properly handles a TokenReader which
returns an EOF in the middle of an open XML
element.

Thanks to Sam Whited for reporting this.

Fixes CVE-2021-27918
Fixes #44913

Change-Id: Id02a3f3def4a1b415fa2d9a8e3b373eb6cb0f433
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/1004594
Reviewed-by: Russ Cox <rsc@google.com>
Reviewed-by: Roland Shoemaker <bracewell@google.com>
Reviewed-by: Filippo Valsorda <valsorda@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/300391
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
encoding/binary: limit bytes read by Uvarint to <= 10 2021-03-08T18:49:14+00:00 Emmanuel T Odeke emmanuel@orijtech.com 2021-03-07T07:14:21+00:00 aafad20b617ee63d58fcd4f6e0d98fe27760678c Limits the number of bytes that can be consumed by Uvarint to MaxVarintLen64 (10) to avoid wasted computations. With this change, if Uvarint reads more than MaxVarintLen64 bytes, it'll return the erroring byte count of n=-(MaxVarintLen64+1) which is -11, as per the function signature. Updated some tests to reflect the new change in expectations of n when the number of bytes to be read exceeds the limits.. Fixes #41185 Change-Id: Ie346457b1ddb0214b60c72e81128e24d604d083d Reviewed-on: https://go-review.googlesource.com/c/go/+/299531 Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Trust: Emmanuel Odeke <emmanuel@orijtech.com> 
Limits the number of bytes that can be consumed by Uvarint
to MaxVarintLen64 (10) to avoid wasted computations.
With this change, if Uvarint reads more than MaxVarintLen64
bytes, it'll return the erroring byte count of n=-(MaxVarintLen64+1)
which is -11, as per the function signature.

Updated some tests to reflect the new change in expectations of n
when the number of bytes to be read exceeds the limits..

Fixes #41185

Change-Id: Ie346457b1ddb0214b60c72e81128e24d604d083d
Reviewed-on: https://go-review.googlesource.com/c/go/+/299531
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
encoding/json: marshal maps using reflect.Value.MapRange 2021-03-05T02:28:11+00:00 Meng Zhuo mzh@golangcn.org 2021-01-21T14:53:30+00:00 a7526bbf72dfe0fde00d88f85fd6cddccdb3a345 Map serialization using reflect.Value.MapIndex cannot retrieve map keys that contain a NaN, resulting in a panic. Switch the implementation to use the reflect.Value.MapRange method instead, which iterates over all map entries regardless of whether they are directly retrievable. Note that according to RFC 8259, section 4, a JSON object should have unique names, but does not forbid the occurrence of duplicate names. Fixes #43207 Change-Id: If4bc55229b1f64b8ca4b0fed37549725efdace39 Reviewed-on: https://go-review.googlesource.com/c/go/+/278632 Trust: Meng Zhuo <mzh@golangcn.org> Trust: Joe Tsai <thebrokentoaster@gmail.com> Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Meng Zhuo <mzh@golangcn.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> 
Map serialization using reflect.Value.MapIndex cannot retrieve
map keys that contain a NaN, resulting in a panic.
Switch the implementation to use the reflect.Value.MapRange method
instead, which iterates over all map entries regardless of whether
they are directly retrievable.

Note that according to RFC 8259, section 4, a JSON object should
have unique names, but does not forbid the occurrence of duplicate names.

Fixes #43207

Change-Id: If4bc55229b1f64b8ca4b0fed37549725efdace39
Reviewed-on: https://go-review.googlesource.com/c/go/+/278632
Trust: Meng Zhuo <mzh@golangcn.org>
Trust: Joe Tsai <thebrokentoaster@gmail.com>
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Meng Zhuo <mzh@golangcn.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>