delta/go-git.git/src/encoding/json/encode_test.go, branch dev.go2go github.com: golang/go encoding/json: detect cyclic maps and slices 2020-09-24T18:18:20+00:00 lujjjh lujjjh@gmail.com 2020-09-17T14:39:13+00:00 428509402b03c608e625a4844ab0cce75e4bead2 Now reports an error if cyclic maps and slices are to be encoded instead of an infinite recursion. This case wasn't handled in CL 187920. Fixes #40745. Change-Id: Ia34b014ecbb71fd2663bb065ba5355a307dbcc15 GitHub-Last-Rev: 6f874944f4065b5237babbb0fdce14c1c74a3c97 GitHub-Pull-Request: golang/go#40756 Reviewed-on: https://go-review.googlesource.com/c/go/+/248358 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Trust: Daniel Martí <mvdan@mvdan.cc> Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Go Bot <gobot@golang.org> 
Now reports an error if cyclic maps and slices are to be encoded
instead of an infinite recursion. This case wasn't handled in CL 187920.

Fixes #40745.

Change-Id: Ia34b014ecbb71fd2663bb065ba5355a307dbcc15
GitHub-Last-Rev: 6f874944f4065b5237babbb0fdce14c1c74a3c97
GitHub-Pull-Request: golang/go#40756
Reviewed-on: https://go-review.googlesource.com/c/go/+/248358
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Trust: Daniel Martí <mvdan@mvdan.cc>
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Go Bot <gobot@golang.org>
Revert "encoding/json: implement Is on all errors" 2020-09-14T21:33:20+00:00 Damien Neil dneil@google.com 2020-09-14T21:00:52+00:00 114719e16e9681bd1001326598ededa719c17944 This reverts CL 254537. Reason for revert: Reason for revert: The recommended way to check for a type of error is errors.As. API changes should also start with a proposal. Change-Id: I07c37428575e99c80b17525833a61831d10963bb Reviewed-on: https://go-review.googlesource.com/c/go/+/254857 Trust: Damien Neil <dneil@google.com> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com> 
This reverts CL 254537.

Reason for revert: Reason for revert: The recommended way to check for a type of error is errors.As. API changes should also start with a proposal.

Change-Id: I07c37428575e99c80b17525833a61831d10963bb
Reviewed-on: https://go-review.googlesource.com/c/go/+/254857
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
encoding/json: implement Is on all errors 2020-09-13T03:19:24+00:00 Carlos Alexandro Becker caarlos0@gmail.com 2020-09-13T02:12:02+00:00 95bb00d1088767ed14e3bd1a5f533a690d619a5f Allows users to check: errors.Is(err, &UnmarshalTypeError{}) errors.Is(err, &UnmarshalFieldError{}) errors.Is(err, &InvalidUnmarshalError{}) errors.Is(err, &UnsupportedValueError{}) errors.Is(err, &MarshalerError{}) which is the recommended way of checking for kinds of errors. SyntaxError.Is was implemented in CL 253037. As and Unwrap relevant methods will be added in future CLs. Change-Id: I1f8a503b8fdc0f3afdfe9669a91f3af8d960e028 GitHub-Last-Rev: 930cda5384c987a0b31f277ba3b4ab690ea74ac3 GitHub-Pull-Request: golang/go#41360 Reviewed-on: https://go-review.googlesource.com/c/go/+/254537 Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com> Trust: Emmanuel Odeke <emm.odeke@gmail.com> 
Allows users to check:

      errors.Is(err, &UnmarshalTypeError{})
      errors.Is(err, &UnmarshalFieldError{})
      errors.Is(err, &InvalidUnmarshalError{})
      errors.Is(err, &UnsupportedValueError{})
      errors.Is(err, &MarshalerError{})

which is the recommended way of checking for kinds of errors.

SyntaxError.Is was implemented in CL 253037.
As and Unwrap relevant methods will be added in future CLs.

Change-Id: I1f8a503b8fdc0f3afdfe9669a91f3af8d960e028
GitHub-Last-Rev: 930cda5384c987a0b31f277ba3b4ab690ea74ac3
GitHub-Pull-Request: golang/go#41360
Reviewed-on: https://go-review.googlesource.com/c/go/+/254537
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
encoding/json: properly encode strings with ",string" again 2020-05-08T13:29:11+00:00 Daniel Martí mvdan@mvdan.cc 2020-03-31T11:20:15+00:00 b1a48af7e8ee87cc46e1bbb07f81ac4853e0f27b golang.org/cl/193604 fixed one bug when one encodes a string with the ",string" option: if SetEscapeHTML(false) is used, we should not be using HTML escaping for the inner string encoding. The CL correctly fixed that. The CL also tried to speed up this edge case. By avoiding an entire new call to Marshal, the new Issue34127 benchmark reduced its time/op by 45%, and lowered the allocs/op from 3 to 2. However, that last optimization wasn't correct: Since Go 1.2 every string can be marshaled to JSON without error even if it contains invalid UTF-8 byte sequences. Therefore there is no need to use Marshal again for the only reason of enclosing the string in double quotes. JSON string encoding isn't just about adding quotes and taking care of invalid UTF-8. We also need to escape some characters, like tabs and newlines. The new code failed to do that. The bug resulted in the added test case failing to roundtrip properly; before our fix here, we'd see an error: invalid use of ,string struct tag, trying to unmarshal "\"\b\f\n\r\t\"\\\"" into string If you pay close attention, you'll notice that the special characters like tab and newline are only encoded once, not twice. When decoding with the ",string" option, the outer string decode works, but the inner string decode fails, as we are now decoding a JSON string with unescaped special characters. The fix we apply here isn't to go back to Marshal, as that would re-introduce the bug with SetEscapeHTML(false). Instead, we can use a new encode state from the pool - it results in minimal performance impact, and even reduces allocs/op further. The performance impact seems fair, given that we need to check the entire string for characters that need to be escaped. name old time/op new time/op delta Issue34127-8 89.7ns ± 2% 100.8ns ± 1% +12.27% (p=0.000 n=8+8) name old alloc/op new alloc/op delta Issue34127-8 40.0B ± 0% 32.0B ± 0% -20.00% (p=0.000 n=8+8) name old allocs/op new allocs/op delta Issue34127-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=8+8) Instead of adding another standalone test, we convert an existing "string tag" test to be table-based, and add another test case there. One test case from the original CL also had to be amended, due to the same problem - when escaping '<' due to SetEscapeHTML(true), we need to end up with double escaping, since we're using ",string". Fixes #38173. Change-Id: I2b0df9e4f1d3452fff74fe910e189c930dde4b5b Reviewed-on: https://go-review.googlesource.com/c/go/+/226498 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> 
golang.org/cl/193604 fixed one bug when one encodes a string with the
",string" option: if SetEscapeHTML(false) is used, we should not be
using HTML escaping for the inner string encoding. The CL correctly
fixed that.

The CL also tried to speed up this edge case. By avoiding an entire new
call to Marshal, the new Issue34127 benchmark reduced its time/op by
45%, and lowered the allocs/op from 3 to 2.

However, that last optimization wasn't correct:

	Since Go 1.2 every string can be marshaled to JSON without error
	even if it contains invalid UTF-8 byte sequences. Therefore
	there is no need to use Marshal again for the only reason of
	enclosing the string in double quotes.

JSON string encoding isn't just about adding quotes and taking care of
invalid UTF-8. We also need to escape some characters, like tabs and
newlines.

The new code failed to do that. The bug resulted in the added test case
failing to roundtrip properly; before our fix here, we'd see an error:

	invalid use of ,string struct tag, trying to unmarshal "\"\b\f\n\r\t\"\\\"" into string

If you pay close attention, you'll notice that the special characters
like tab and newline are only encoded once, not twice. When decoding
with the ",string" option, the outer string decode works, but the inner
string decode fails, as we are now decoding a JSON string with unescaped
special characters.

The fix we apply here isn't to go back to Marshal, as that would
re-introduce the bug with SetEscapeHTML(false). Instead, we can use a
new encode state from the pool - it results in minimal performance
impact, and even reduces allocs/op further. The performance impact seems
fair, given that we need to check the entire string for characters that
need to be escaped.

	name          old time/op    new time/op    delta
	Issue34127-8    89.7ns ± 2%   100.8ns ± 1%  +12.27%  (p=0.000 n=8+8)

	name          old alloc/op   new alloc/op   delta
	Issue34127-8     40.0B ± 0%     32.0B ± 0%  -20.00%  (p=0.000 n=8+8)

	name          old allocs/op  new allocs/op  delta
	Issue34127-8      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=8+8)

Instead of adding another standalone test, we convert an existing
"string tag" test to be table-based, and add another test case there.

One test case from the original CL also had to be amended, due to the
same problem - when escaping '<' due to SetEscapeHTML(true), we need to
end up with double escaping, since we're using ",string".

Fixes #38173.

Change-Id: I2b0df9e4f1d3452fff74fe910e189c930dde4b5b
Reviewed-on: https://go-review.googlesource.com/c/go/+/226498
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
encoding/json: error when encoding a pointer cycle 2019-11-11T16:24:21+00:00 Daniel Martí mvdan@mvdan.cc 2019-07-29T03:16:14+00:00 64c9ee98b7684cf2156f620cbab4dbb6081b9771 Otherwise we'd panic with a stack overflow. Most programs are in control of the data being encoded and can ensure there are no cycles, but sometimes it's not that simple. For example, running a user's html template with script tags can easily result in crashes if the user can find a pointer cycle. Adding the checks via a map to every ptrEncoder.encode call slowed down the benchmarks below by a noticeable 13%. Instead, only start doing the relatively expensive pointer cycle checks if we're many levels of pointers deep in an encode state. A threshold of 1000 is small enough to capture pointer cycles before they're a problem (the goroutine stack limit is currently 1GB, and I needed close to a million levels to reach it). Yet it's large enough that reasonable uses of the json encoder only see a tiny 1% slow-down due to the added ptrLevel field and check. name old time/op new time/op delta CodeEncoder-8 2.34ms ± 1% 2.37ms ± 0% +1.05% (p=0.000 n=10+10) CodeMarshal-8 2.42ms ± 1% 2.44ms ± 0% +1.10% (p=0.000 n=10+10) name old speed new speed delta CodeEncoder-8 829MB/s ± 1% 820MB/s ± 0% -1.04% (p=0.000 n=10+10) CodeMarshal-8 803MB/s ± 1% 795MB/s ± 0% -1.09% (p=0.000 n=10+10) name old alloc/op new alloc/op delta CodeEncoder-8 43.1kB ± 8% 42.5kB ±10% ~ (p=0.989 n=10+10) CodeMarshal-8 1.99MB ± 0% 1.99MB ± 0% ~ (p=0.254 n=9+6) name old allocs/op new allocs/op delta CodeEncoder-8 0.00 0.00 ~ (all equal) CodeMarshal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Finally, add a few tests to ensure that the code handles the edge cases properly. Fixes #10769. Change-Id: I73d48e0cf6ea140127ea031f2dbae6e6a55e58b8 Reviewed-on: https://go-review.googlesource.com/c/go/+/187920 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com> Reviewed-by: Andrew Bonventre <andybons@golang.org> 
Otherwise we'd panic with a stack overflow.

Most programs are in control of the data being encoded and can ensure
there are no cycles, but sometimes it's not that simple. For example,
running a user's html template with script tags can easily result in
crashes if the user can find a pointer cycle.

Adding the checks via a map to every ptrEncoder.encode call slowed down
the benchmarks below by a noticeable 13%. Instead, only start doing the
relatively expensive pointer cycle checks if we're many levels of
pointers deep in an encode state.

A threshold of 1000 is small enough to capture pointer cycles before
they're a problem (the goroutine stack limit is currently 1GB, and I
needed close to a million levels to reach it). Yet it's large enough
that reasonable uses of the json encoder only see a tiny 1% slow-down
due to the added ptrLevel field and check.

	name           old time/op    new time/op    delta
	CodeEncoder-8    2.34ms ± 1%    2.37ms ± 0%  +1.05%  (p=0.000 n=10+10)
	CodeMarshal-8    2.42ms ± 1%    2.44ms ± 0%  +1.10%  (p=0.000 n=10+10)

	name           old speed      new speed      delta
	CodeEncoder-8   829MB/s ± 1%   820MB/s ± 0%  -1.04%  (p=0.000 n=10+10)
	CodeMarshal-8   803MB/s ± 1%   795MB/s ± 0%  -1.09%  (p=0.000 n=10+10)

	name           old alloc/op   new alloc/op   delta
	CodeEncoder-8    43.1kB ± 8%    42.5kB ±10%    ~     (p=0.989 n=10+10)
	CodeMarshal-8    1.99MB ± 0%    1.99MB ± 0%    ~     (p=0.254 n=9+6)

	name           old allocs/op  new allocs/op  delta
	CodeEncoder-8      0.00           0.00         ~     (all equal)
	CodeMarshal-8      1.00 ± 0%      1.00 ± 0%    ~     (all equal)

Finally, add a few tests to ensure that the code handles the edge cases
properly.

Fixes #10769.

Change-Id: I73d48e0cf6ea140127ea031f2dbae6e6a55e58b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/187920
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
encoding/json: correct caller's name in encoding errors 2019-10-16T22:58:08+00:00 Eugene Kalinin e.v.kalinin@gmail.com 2019-06-28T20:59:49+00:00 02196d36575636a64f868ee0ffe6bb61442e7245 1. Change mapencode.encode to use fmt.Error rather than MarshalerError. MarshalerError refer to MarshalJSON, but mapencode.encode does not use that. 2. Add sourceFunc field to MarshalerError to record the name of the function that creates the error, so that the Error method can report it correctly. Fixes #29753 Change-Id: I186c2fac8470ae2f9e300501de3730face642230 Reviewed-on: https://go-review.googlesource.com/c/go/+/184119 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> 
1. Change mapencode.encode to use fmt.Error rather than MarshalerError.
MarshalerError refer to MarshalJSON, but mapencode.encode does not use that.

2. Add sourceFunc field to MarshalerError to record the name of the function
that creates the error, so that the Error method can report it correctly.

Fixes #29753

Change-Id: I186c2fac8470ae2f9e300501de3730face642230
Reviewed-on: https://go-review.googlesource.com/c/go/+/184119
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
encoding/json: make Number with the ,string option marshal with quotes 2019-09-16T11:56:15+00:00 Lucas Bremgartner lucas@bremis.ch 2019-09-13T19:46:50+00:00 49e7c7672d6d065435f7058df90b082cb552c7dd Add quotes when marshaling a json.Number with the string option set via a struct tag. This ensures that the resulting json can be unmarshaled into the source struct without error. Fixes #34268 Change-Id: Ide167d9dec77019554870b5957b37dc258119d81 GitHub-Last-Rev: dde81b71208be01c253bb87dbb6f81ac6e0785be GitHub-Pull-Request: golang/go#34269 Reviewed-on: https://go-review.googlesource.com/c/go/+/195043 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> 
Add quotes when marshaling a json.Number with the string option
set via a struct tag. This ensures that the resulting json
can be unmarshaled into the source struct without error.

Fixes #34268

Change-Id: Ide167d9dec77019554870b5957b37dc258119d81
GitHub-Last-Rev: dde81b71208be01c253bb87dbb6f81ac6e0785be
GitHub-Pull-Request: golang/go#34269
Reviewed-on: https://go-review.googlesource.com/c/go/+/195043
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
encoding/json: encode nil encoding.TextMarshaler instance as "null" 2019-09-11T18:37:43+00:00 William Poussier william.poussier@gmail.com 2019-09-11T15:33:25+00:00 cc39d8087b20fce14e60b7e5f287593da2c72749 Fixes #34235. Change-Id: Ia3795fd18860530fa6a4b171545f525e784ffdcb GitHub-Last-Rev: 1a319c452857818f7aaf22ef46823b43ca9b2276 GitHub-Pull-Request: golang/go#34238 Reviewed-on: https://go-review.googlesource.com/c/go/+/194642 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> 
Fixes #34235.

Change-Id: Ia3795fd18860530fa6a4b171545f525e784ffdcb
GitHub-Last-Rev: 1a319c452857818f7aaf22ef46823b43ca9b2276
GitHub-Pull-Request: golang/go#34238
Reviewed-on: https://go-review.googlesource.com/c/go/+/194642
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
src: gofmt -s 2019-09-09T18:57:05+00:00 Ainar Garipov gugl.zadolbal@gmail.com 2019-09-09T06:00:53+00:00 51c8d969bd8a9992002a11373f91564c326e6d77 Change-Id: I56d7eeaf777ac30886ee77428ca1ac72b77fbf7d Reviewed-on: https://go-review.googlesource.com/c/go/+/193849 Run-TryBot: Dave Cheney <dave@cheney.net> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> 
Change-Id: I56d7eeaf777ac30886ee77428ca1ac72b77fbf7d
Reviewed-on: https://go-review.googlesource.com/c/go/+/193849
Run-TryBot: Dave Cheney <dave@cheney.net>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
encoding/json: fix panic for nil instances of TextMarshaler in map keys 2019-09-01T16:39:38+00:00 William Poussier william.poussier@gmail.com 2019-09-01T15:38:31+00:00 85f3ca74882703042bcc2f17353eda192d649920 This change adds a a check in the encodeWithString.resolve method to ensure that a reflect.Value with kind Ptr is not nil before the type assertion to TextMarshaler. If the value is nil, the method returns a nil error, and the map key encodes to an empty string. Fixes #33675 Change-Id: I0a04cf690ae67006f6a9c5f8cbb4cc99d236bca8 GitHub-Last-Rev: 6c987c90846f854e21814dbfb3a073605ec8a94c GitHub-Pull-Request: golang/go#33700 Reviewed-on: https://go-review.googlesource.com/c/go/+/190697 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> 
This change adds a a check in the encodeWithString.resolve method
to ensure that a reflect.Value with kind Ptr is not nil before
the type assertion to TextMarshaler.

If the value is nil, the method returns a nil error, and the map key
encodes to an empty string.

Fixes #33675

Change-Id: I0a04cf690ae67006f6a9c5f8cbb4cc99d236bca8
GitHub-Last-Rev: 6c987c90846f854e21814dbfb3a073605ec8a94c
GitHub-Pull-Request: golang/go#33700
Reviewed-on: https://go-review.googlesource.com/c/go/+/190697
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>