delta/go-git.git/src/runtime/export_test.go, branch dev.boringcrypto

runtime: refactor the scavenger and make it testable

2022-04-26T22:15:21+00:00

This change refactors the scavenger into a type whose methods represent
the actual function and scheduling of the scavenger. It also stubs out
access to global state in order to make it testable.

This change thus also adds a test for the scavenger. In writing this
test, I discovered the lack of a behavior I expected: if the
pageAlloc.scavenge returns < the bytes requested scavenged, that means
the heap is exhausted. This has been true this whole time, but was not
documented or explicitly relied upon. This change rectifies that. In
theory this means the scavenger could spin in run() indefinitely (as
happened in the test) if shouldStop never told it to stop. In practice,
shouldStop fires long before the heap is exhausted, but for future
changes it may be important. At the very least it's good to be
intentional about these things.

While we're here, I also moved the call to stopTimer out of wake and
into sleep. There's no reason to add more operations to a context that's
already precarious (running without a P on sysmon).

Change-Id: Ib31b86379fd9df84f25ae282734437afc540da5c
Reviewed-on: https://go-review.googlesource.com/c/go/+/384734
Reviewed-by: Michael Pratt 
Run-TryBot: Michael Knyszek 
TryBot-Result: Gopher Robot

runtime: reduce max idle mark workers during periodic GC cycles

2022-04-26T22:08:42+00:00

This change reduces the maximum number of idle mark workers during
periodic (currently every 2 minutes) GC cycles to 1.

Idle mark workers soak up all available and unused Ps, up to GOMAXPROCS.
While this provides some throughput and latency benefit in general, it
can cause what appear to be massive CPU utilization spikes in otherwise
idle applications. This is mostly an issue for *very* idle applications,
ones idle enough to trigger periodic GC cycles. This spike also tends to
interact poorly with auto-scaling systems, as the system might assume
the load average is very low and suddenly see a massive burst in
activity.

The result of this change is not to bring down this 100% (of GOMAXPROCS)
CPU utilization spike to 0%, but rather

  min(25% + 1/GOMAXPROCS*100%, 100%)

Idle mark workers also do incur a small latency penalty as they must be
descheduled for other work that might pop up. Luckily the runtime is
pretty good about getting idle mark workers off of Ps, so in general
the latency benefit from shorter GC cycles outweighs this cost. But, the
cost is still non-zero and may be more significant in idle applications
that aren't invoking assists and write barriers quite as often.

We can't completely eliminate idle mark workers because they're
currently necessary for GC progress in some circumstances. Namely,
they're critical for progress when all we have is fractional workers. If
a fractional worker meets its quota, and all user goroutines are blocked
directly or indirectly on a GC cycle (via runtime.GOMAXPROCS, or
runtime.GC), the program may deadlock without GC workers, since the
fractional worker will go to sleep with nothing to wake it.

Fixes #37116.
For #44163.

Change-Id: Ib74793bb6b88d1765c52d445831310b0d11ef423
Reviewed-on: https://go-review.googlesource.com/c/go/+/393394
Reviewed-by: Michael Pratt 
Run-TryBot: Michael Knyszek 
TryBot-Result: Gopher Robot

runtime: fix stack-move sensitivity in some tests

2022-04-25T18:06:52+00:00

There are a few tests of the scheduler run queue API that allocate a
local []g and test using those G's. However, the run queue API
frequently converts between *g and guintptr, which is safe for "real"
Gs because they're heap-allocated and hence don't move, but if these
tests get a stack movement while holding one of these local *g's as a
guintptr, it won't get updated and the test will fail.

Updates #48297.

Change-Id: Ifd424147ce1a1b53732ff0cf55a81df1a9beeb3b
Reviewed-on: https://go-review.googlesource.com/c/go/+/402157
Run-TryBot: Austin Clements 
TryBot-Result: Gopher Robot 
Reviewed-by: Cherry Mui

runtime: add fastrand64

2022-04-20T22:50:33+00:00

Support fastrand64 in the runtime, although fastrand uses wyrand to generate 64-bit random number, it still returns uint32. In some cases, we need to generate a 64-bit random number, the new API would be faster and easier to use, and at least we can use the new function in these places:

src/net/dnsclient.go:randInt()
src/hash/maphash/maphash.go:MakeSeed()
src/runtime/map.go:mapiterinit()

name                 time/op
Fastrand-16          0.09ns ± 5%
Fastrand64-16        0.09ns ± 6%

Change-Id: Ibb97378c7ca59bc7dc15535d4872fa58ea112e6a
Reviewed-on: https://go-review.googlesource.com/c/go/+/400734
Reviewed-by: Keith Randall 
Run-TryBot: Keith Randall 
Auto-Submit: Keith Randall 
Reviewed-by: Keith Randall 
TryBot-Result: Gopher Robot 
Reviewed-by: Ian Lance Taylor

all: separate doc comment from //go: directives

2022-04-05T17:54:15+00:00

A future change to gofmt will rewrite

	// Doc comment.
	//go:foo

to

	// Doc comment.
	//
	//go:foo

Apply that change preemptively to all comments (not necessarily just doc comments).

For #51082.

Change-Id: Iffe0285418d1e79d34526af3520b415a12203ca9
Reviewed-on: https://go-review.googlesource.com/c/go/+/384260
Trust: Russ Cox 
Run-TryBot: Russ Cox 
Reviewed-by: Ian Lance Taylor 
TryBot-Result: Gopher Robot

runtime: remove old pacer and the PacerRedesign goexperiment

2022-03-31T20:02:10+00:00

Now that Go 1.18 has been released, remove the old pacer.

Change-Id: Ie7a7596d67f3fc25d3f375a08fc75eafac2eb834
Reviewed-on: https://go-review.googlesource.com/c/go/+/393396
Trust: Michael Knyszek 
Reviewed-by: Austin Clements 
Reviewed-by: Michael Pratt

runtime: make piController much more defensive about overflow

2022-02-10T18:55:42+00:00

If something goes horribly wrong with the assumptions surrounding a
piController, its internal error state might accumulate in an unbounded
manner. In practice this means unexpected Inf and NaN values.

Avoid this by identifying cases where the error overflows and resetting
controller state.

In the scavenger, this case is much more likely. All that has to happen
is the proportional relationship between sleep time and estimated CPU
usage has to break down. Unfortunately because we're just measuring
monotonic time for all this, there are lots of ways it could happen,
especially in an oversubscribed system. In these cases, just fall back
on a conservative pace for scavenging and try to wait out the issue.

In the pacer I'm pretty sure this is impossible. Because we wire the
output of the controller to the input, the response is very directly
correlated, so it's impossible for the controller's core assumption to
break down.

While we're in the pacer, add more detail about why that controller is
even there, as well as its purpose.

Finally, let's be proactive about other sources of overflow, namely
overflow from a very large input value. This change adds a check after
the first few operations to detect overflow issues from the input,
specifically the multiplication.

No tests for the pacer because I was unable to actually break the
pacer's controller under a fuzzer, and no tests for the scavenger because
it is not really in a testable state.

However:
* This change includes a fuzz test for the piController.
* I broke out the scavenger code locally and fuzz tested it, confirming
  that the patch eliminates the original failure mode.
* I tested that on a local heap-spike test, the scavenger continues
  operating as expected under normal conditions.

Fixes #51061.

Change-Id: I02a01d2dbf0eb9d2a8a8e7274d4165c2b6a3415a
Reviewed-on: https://go-review.googlesource.com/c/go/+/383954
Reviewed-by: David Chase 
Reviewed-by: Michael Pratt 
Trust: Michael Knyszek 
Run-TryBot: Michael Knyszek 
TryBot-Result: Gopher Robot

runtime: simplify histogram buckets considerably

2022-02-10T00:07:14+00:00

There was an off-by-one error in the time histogram buckets calculation
that caused the linear sub-buckets distances to be off by 2x.

The fix was trivial, but in writing tests I realized there was a much
simpler way to express the calculation for the histogram buckets, and
took the opportunity to do that here. The new bucket calculation also
fixes the bug.

Fixes #50732.

Change-Id: Idae89986de1c415ee4e148f778e0e101ca003ade
Reviewed-on: https://go-review.googlesource.com/c/go/+/380094
Reviewed-by: Michael Pratt 
Reviewed-by: Emmanuel Odeke 
Trust: Michael Knyszek 
Run-TryBot: Michael Knyszek

runtime: replace TestFutexsleep with TestTimediv

2022-01-24T17:50:23+00:00

TestFutexsleep was originally created in CL 7876043 as a
regression test for buggy division logic in futexsleep. Several months
later CL 11575044 moved this logic to timediv (called by futexsleep).

This test calls runtime.Futexsleep, which temporarily disables
asynchronous preemption. Unfortunately, TestFutexSleep calls this from
multiple goroutines, creating a race condition that may result in
asynchronous preemption remaining disabled for the remainder of the
process lifetime.

We could fix this by moving the async preemption disable to the main
test function, however this test has had a history of flakiness. As an
alternative, this CL replaces the test wholesale with a new test for
timediv, covering the overflow logic without the difficulty of dealing
with futex.

Fixes #50749.

Change-Id: If9e1dac63ef1535adb49f9a9ffcaff99b9135895
Reviewed-on: https://go-review.googlesource.com/c/go/+/380058
Trust: Michael Pratt 
Run-TryBot: Michael Pratt 
TryBot-Result: Gopher Robot 
Reviewed-by: Cherry Mui 
Reviewed-by: Michael Knyszek

all: gofmt -w -r 'interface{} -> any' src

2021-12-13T18:45:54+00:00

And then revert the bootstrap cmd directories and certain testdata.
And adjust tests as needed.

Not reverting the changes in std that are bootstrapped,
because some of those changes would appear in API docs,
and we want to use any consistently.
Instead, rewrite 'any' to 'interface{}' in cmd/dist for those directories
when preparing the bootstrap copy.

A few files changed as a result of running gofmt -w
not because of interface{} -> any but because they
hadn't been updated for the new //go:build lines.

Fixes #49884.

Change-Id: Ie8045cba995f65bd79c694ec77a1b3d1fe01bb09
Reviewed-on: https://go-review.googlesource.com/c/go/+/368254
Trust: Russ Cox 
Run-TryBot: Russ Cox 
Reviewed-by: Robert Griesemer 
TryBot-Result: Gopher Robot