summaryrefslogtreecommitdiff
path: root/src/go/token
Commit message (Collapse)AuthorAgeFilesLines
* go/token: use atomics not Mutex for last file cacheAlan Donovan2022-06-292-16/+20
| | | | | | | | | | | | | | | | | | | Previously, FileSet would cache the last *File found by a lookup, using a full (exclusive) mutex within FileSet.File, turning a logical read operation into an update. This was one of the largest sources of contention in gopls. This change uses atomic load/store on the 'last' field without a mutex. Also, in FileSet.AddFile, allocate the File outside the critical section; all the other operations are typically cheap. Fixes #53507 Change-Id: Ice8641650d8495b25b0428e9b9320837ff2ca7e1 Reviewed-on: https://go-review.googlesource.com/c/go/+/411909 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Robert Griesemer <gri@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
* go/token: delete unused File.set fieldAlan Donovan2022-06-173-9/+1
| | | | | | | | | | | This field is only used for a sanity check in a test. Change-Id: I868ed10131ec33994ebb1b1d88f6740956824bd7 Reviewed-on: https://go-review.googlesource.com/c/go/+/409834 Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Robert Griesemer <gri@google.com>
* all: gofmt main repoRuss Cox2022-04-111-1/+0
| | | | | | | | | | | | | | | [This CL is part of a sequence implementing the proposal #51082. The design doc is at https://go.dev/s/godocfmt-design.] Run the updated gofmt, which reformats doc comments, on the main repository. Vendored files are excluded. For #51082. Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407 Reviewed-on: https://go-review.googlesource.com/c/go/+/384268 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
* all: remove trailing blank doc comment linesRuss Cox2022-04-012-33/+0
| | | | | | | | | | | | | | | | | | | | | | | | A future change to gofmt will rewrite // Doc comment. // func f() to // Doc comment. func f() Apply that change preemptively to all doc comments. For #51082. Change-Id: I4023e16cfb0729b64a8590f071cd92f17343081d Reviewed-on: https://go-review.googlesource.com/c/go/+/384259 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
* go/token: allocate fewer times at init timeDaniel Martí2022-03-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | go/token has had a global "keywords" map filled at init time for years. Overall, the package's init time cost is small, as per GODEBUG=inittrace=1: init go/token @0.51 ms, 0.004 ms clock, 1776 bytes, 5 allocs init go/token @0.44 ms, 0.003 ms clock, 1776 bytes, 5 allocs init go/token @0.45 ms, 0.003 ms clock, 1568 bytes, 4 allocs However, adding the map size hint does help with the allocations: init go/token @0.45 ms, 0.002 ms clock, 944 bytes, 2 allocs init go/token @0.46 ms, 0.002 ms clock, 944 bytes, 2 allocs init go/token @0.55 ms, 0.003 ms clock, 1152 bytes, 3 allocs Three samples are rather unscientific, and the clock time is basically unchanged, but we might as well reduce the allocs. Change-Id: I48121a4cea4113d991882e32f274d7b7736800dc Reviewed-on: https://go-review.googlesource.com/c/go/+/391094 Trust: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org>
* go/token: slight performance improvement for IsIdentifierchanxuehong2022-03-021-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | If name is empty or a keyword, we can skip the loop entirely. Otherwise, we do the same amount of work as before. Here is the benchmark result for go/parser: name old time/op new time/op delta Parse-12 2.53ms ± 2% 2.47ms ± 1% -2.38% (p=0.000 n=9+10) ParseOnly-12 1.97ms ± 1% 1.93ms ± 2% -1.80% (p=0.000 n=10+10) Resolve-12 560µs ± 1% 558µs ± 1% ~ (p=0.200 n=9+8) name old speed new speed delta Parse-12 26.1MB/s ± 2% 26.8MB/s ± 1% +2.44% (p=0.000 n=9+10) ParseOnly-12 33.6MB/s ± 1% 34.3MB/s ± 2% +1.82% (p=0.000 n=10+10) Resolve-12 118MB/s ± 2% 119MB/s ± 1% ~ (p=0.116 n=10+8) Change-Id: I87ac9c2637a6c0e697382b74245ac88ef523bba7 GitHub-Last-Rev: 036bc38d837c095dd5a8d97ece83e1596d875d3e GitHub-Pull-Request: golang/go#48534 Reviewed-on: https://go-review.googlesource.com/c/go/+/351389 Trust: David Chase <drchase@google.com> Trust: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Robert Griesemer <gri@golang.org>
* all: gofmt -w -r 'interface{} -> any' srcRuss Cox2021-12-132-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | And then revert the bootstrap cmd directories and certain testdata. And adjust tests as needed. Not reverting the changes in std that are bootstrapped, because some of those changes would appear in API docs, and we want to use any consistently. Instead, rewrite 'any' to 'interface{}' in cmd/dist for those directories when preparing the bootstrap copy. A few files changed as a result of running gofmt -w not because of interface{} -> any but because they hadn't been updated for the new //go:build lines. Fixes #49884. Change-Id: Ie8045cba995f65bd79c694ec77a1b3d1fe01bb09 Reviewed-on: https://go-review.googlesource.com/c/go/+/368254 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
* go/token: match the implementation of index selection with sort.Searchsubham sarkar2021-08-241-1/+1
| | | | | | | | | | | | | | | name old time/op new time/op delta SearchInts-8 15.5ns ± 2% 13.7ns ± 4% -11.87% (p=0.008 n=5+5) (see CL 36332 for the original change to sort.Search) Change-Id: If452818185b92b8b3548b066f475e493d604ea29 GitHub-Last-Rev: 32dd3cffa6b54b332948ac6a2929458defd4838f GitHub-Pull-Request: golang/go#47293 Reviewed-on: https://go-review.googlesource.com/c/go/+/335809 Reviewed-by: Robert Griesemer <gri@golang.org> Trust: Robert Griesemer <gri@golang.org> Trust: Than McIntosh <thanm@google.com>
* [dev.typeparams] go/token, go/scanner: add the "~" operatorRob Findley2021-06-171-1/+10
| | | | | | | | | | | This is an approximate port of CL 307370 to go/token and go/scanner. Change-Id: I5b789408f825f7e39f569322cb67802117b9d734 Reviewed-on: https://go-review.googlesource.com/c/go/+/324992 Trust: Robert Findley <rfindley@google.com> Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org>
* go/token: correct the interval notation used in some panic messagesRob Findley2021-05-171-2/+2
| | | | | | | | | | | | | | | Fix an apparent typo for the right-hand bound in a couple panic messages, where '[' was used instead of ']'. Fixes #46215 Change-Id: Ie419c404ca72ed085a83a2c38ea1a5d6ed326cca Reviewed-on: https://go-review.googlesource.com/c/go/+/320510 Trust: Robert Findley <rfindley@google.com> Trust: Robert Griesemer <gri@golang.org> Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org>
* all: faster midpoint computation in binary searchyangwenmai2021-02-232-1/+25
| | | | | | | | | | | | | | On my machine (3.1 GHz Quad-Core Intel Core i7, macOS 10.15.7 10.15.7), go 1.15.6 benchstat: name old time/op new time/op delta SearchInts-8 20.3ns ± 1% 16.6ns ± 6% -18.37% (p=0.000 n=9+10) Change-Id: I346e5955fd6df6ce10254b22267dbc8d5a2b16c0 Reviewed-on: https://go-review.googlesource.com/c/go/+/279439 Reviewed-by: Ben Shi <powerman1st@163.com> Reviewed-by: Robert Griesemer <gri@golang.org> Trust: Robert Griesemer <gri@golang.org>
* token: more descriptive panicsJean de Klerk2020-10-141-9/+12
| | | | | | | | | | | | | | | | Currently, there are several panics in token that simply say "illegal!". This CL adds the values. This is valuable when the token call is wrapped under several layers and you can't easily see which value is being passed to token. Change-Id: Ib04b55cafcd9b9ec6820dcf416fc4d49afaea15f Reviewed-on: https://go-review.googlesource.com/c/go/+/262017 Reviewed-by: Robert Griesemer <gri@golang.org> Run-TryBot: Robert Griesemer <gri@golang.org> Trust: Robert Griesemer <gri@golang.org> Trust: Jean de Klerk <deklerk@google.com> TryBot-Result: Go Bot <gobot@golang.org>
* go/token: explain file base offset better in documentationRobert Griesemer2020-06-171-2/+21
| | | | | | | | Fixes #36648. Change-Id: I92d4462fea0079f63697fb8f407fd2d50b7d68f7 Reviewed-on: https://go-review.googlesource.com/c/go/+/238117 Reviewed-by: Ian Lance Taylor <iant@golang.org>
* go/token: add IsIdentifier, IsKeyword, and IsExportedDaniel Martí2019-04-152-1/+66
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Telling whether a string is a valid Go identifier can seem like an easy task, but it's easy to forget about the edge cases. For example, some implementations out there forget that an empty string or keywords like "func" aren't valid identifiers. Add a simple implementation with proper Unicode support, and start using it in cmd/cover and cmd/doc. Other pieces of the standard library reimplement part of this logic, but don't use a "func(string) bool" signature, so we're leaving them untouched for now. Add some tests too, to ensure that we actually got these edge cases correctly. Since telling whether a string is a valid identifier requires knowing that it's not a valid keyword, add IsKeyword too. The internal map was already accessible via Lookup, but "Lookup(str) != IDENT" isn't as easy to understand as IsKeyword(str). And, as per Josh's suggestion, we could have IsKeyword (and probably Lookup too) use a perfect hash function instead of a global map. Finally, for consistency with these new functions, add IsExported. That makes go/ast.IsExported a bit redundant, so perhaps it can be deprecated in favor of go/token.IsExported in the future. Clarify that token.IsExported doesn't imply token.IsIdentifier, to avoid ambiguity. Fixes #30064. Change-Id: I0e0e49215fd7e47b603ebc2b5a44086c51ba57f7 Reviewed-on: https://go-review.googlesource.com/c/go/+/169018 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org> Reviewed-by: Alan Donovan <adonovan@google.com>
* go/token: add (*File).LineStart, which returns Pos for a given lineAlan Donovan2018-09-102-1/+31
| | | | | | | | | | | | | LineStart returns the position of the start of a given line. Like MergeLine, it panics if the 1-based line number is invalid. This function is especially useful in programs that occasionally handle non-Go files such as assembly but wish to use the token.Pos mechanism to identify file positions. Change-Id: I5f774c0690074059553cdb38c0f681f5aafc8da1 Reviewed-on: https://go-review.googlesource.com/134075 Reviewed-by: Robert Griesemer <gri@golang.org>
* go/token: add example for retrieving Position from Posjimmyfrasche2018-03-131-0/+77
| | | | | | | | | | | | | | | | There are few uses for the majority of the API in go/token for the average user. The exception to this is getting the filename, line, and column information from a token.Pos (reported and absolute. This is straightforward but figuring out how to do it requires combing through a lot of documentation. This example makes it more easily discoverable. Updates #24352. Change-Id: I0a45da6173b3dabebf42484bbbed30d9e5e20e01 Reviewed-on: https://go-review.googlesource.com/100058 Reviewed-by: Robert Griesemer <gri@golang.org> Run-TryBot: Robert Griesemer <gri@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* go/scanner: recognize //line and /*line directives incl. columnsRobert Griesemer2018-03-091-17/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change updates go/scanner to recognize the extended line directives that are now also handled by cmd/compile: //line filename:line //line filename:line:column /*line filename:line*/ /*line filename:line:column*/ As before, //-style line directives must start in column 1. /*-style line directives may be placed anywhere in the code. In both cases, the specified position applies to the character immediately following the comment; for line comments that is the first character on the next line (after the newline of the comment). The go/token API is extended by a new method File.AddLineColumnInfo(offset int, filename string, line, column int) which extends the existing File.AddLineInfo(offset int, filename string, line int) by adding a column parameter. Adjusted token.Position computation is changed to take into account column information if provided via a line directive: A (line-directive) relative position will have a non-zero column iff the line directive specified a column; if the position is on the same line as the line directive, the column is relative to the specified column (otherwise it is relative to the line beginning). See also #24183. Finally, Position.String() has been adjusted to not print a column value if the column is unknown (== 0). Fixes #24143. Change-Id: I5518c825ad94443365c049a95677407b46ba55a1 Reviewed-on: https://go-review.googlesource.com/97795 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
* go/token: use fine-grained locking in FileSetAlan Donovan2017-06-142-19/+35
| | | | | | | | | | | | | | | | | Before, all accesses to the lines and infos tables of each File were serialized by the lock of the owning FileSet, causing parsers running in parallel to contend. Now, each File has its own mutex. This fixes a data race in (*File).PositionFor, which used to call f.position then f.unpack without holding the mutex's lock. Fixes golang/go#18348 Change-Id: Iaa5989b2eba88a7fb2e91c1a0a8bc1e7f6497f2b Reviewed-on: https://go-review.googlesource.com/34591 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* go/token: remove excess parenthesis in NoPos.IsValid() documentationIbrahim AshShohail2017-05-091-1/+1
| | | | | | | | Fixes #20294 Change-Id: I32ac862fe00180210a04103cc94c4d9fef5d1b6c Reviewed-on: https://go-review.googlesource.com/42992 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* Revert "go/scanner, go/token: recognize => (ALIAS) token"Robert Griesemer2016-11-051-7/+1
| | | | | | | | | | | | This reverts commit 776a90100f1f65fcf54dfd3d082d657341bdc323. Reason: Decision to back out current alias implementation. For #16339. Change-Id: Icb451a122c661ded05d9293356b466fa72b965f3 Reviewed-on: https://go-review.googlesource.com/32824 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
* go/scanner, go/token: recognize => (ALIAS) tokenRobert Griesemer2016-10-041-1/+7
| | | | | | | | For #16339. Change-Id: I0f83e46f13b5c8801aacf48fc8b690049edbbbff Reviewed-on: https://go-review.googlesource.com/30210 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
* go/token: Fix race in FileSet.PositionFor.Jan Mercl2016-08-162-1/+32
| | | | | | | | | | | | | | | | | | | | Methods of FileSet are documented to be safe for concurrent use by multiple goroutines, so FileSet is protected by a mutex and all its methods use it to prevent concurrent mutations. All methods of File that mutate the respective FileSet, including AddLine, do also lock its mutex, but that does not help when PositionFor is invoked concurrently and reads without synchronization what AddLine mutates. The change adds acquiring a RLock around the racy call of File.position and the respective test. Fixes #16548 Change-Id: Iecaaa02630b2532cb29ab555376633ee862315dd Reviewed-on: https://go-review.googlesource.com/25345 Reviewed-by: Robert Griesemer <gri@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
* go/token: document postcondition of SetLinesAlan Donovan2016-05-051-0/+1
| | | | | | Change-Id: Ie163deade396b3e298a93845b9ca4d52333ea82a Reviewed-on: https://go-review.googlesource.com/22831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
* all: use "reports whether" in place of "returns true if(f)"Josh Bleecher Snyder2015-03-181-3/+3
| | | | | | | | Comment changes only. Change-Id: I56848814564c4aa0988b451df18bebdfc88d6d94 Reviewed-on: https://go-review.googlesource.com/7721 Reviewed-by: Rob Pike <r@golang.org>
* go/token: document that column positions and file offsets are in bytesRobert Griesemer2015-02-231-3/+3
| | | | | | | | Fixes #9948. Change-Id: I7b354fccd5e933eeeb2253a66acec050ebff6e41 Reviewed-on: https://go-review.googlesource.com/5611 Reviewed-by: Alan Donovan <adonovan@google.com>
* build: move package sources from src/pkg to srcRuss Cox2014-09-085-0/+1257
Preparation was in CL 134570043. This CL contains only the effect of 'hg mv src/pkg/* src'. For more about the move, see golang.org/s/go14nopkg.
View Lorry Controller Status