summaryrefslogtreecommitdiff
path: root/Lib/lib2to3/pgen2/tokenize.py
Commit message (Collapse)AuthorAgeFilesLines
* Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-1/+1
|
* Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-14/+19
| | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-1/+11
| | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-111-2/+61
|
* add matrix multiplication operator support to 2to3Benjamin Peterson2014-04-101-1/+1
|
* Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-0/+3
| | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* #19620: Fix typo in docstring (noticed by Christopher Welborn).Ezio Melotti2013-11-251-2/+2
|
* Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py scriptSerhiy Storchaka2013-09-161-5/+4
| | | | now detect Python source code encoding only in comment lines.
* Merged revisions 80934 via svnmerge fromBenjamin Peterson2010-05-071-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ................ r80934 | benjamin.peterson | 2010-05-07 13:58:23 -0500 (Fri, 07 May 2010) | 69 lines Merged revisions 79911,79916-79917,80018,80418,80572-80573,80635-80639,80668,80922 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r79911 | benjamin.peterson | 2010-04-09 15:38:53 -0500 (Fri, 09 Apr 2010) | 1 line use absolute import ........ r79916 | benjamin.peterson | 2010-04-09 16:05:21 -0500 (Fri, 09 Apr 2010) | 1 line generalize detection of __future__ imports and attach them to the tree ........ r79917 | benjamin.peterson | 2010-04-09 16:11:44 -0500 (Fri, 09 Apr 2010) | 1 line don't try to 'fix' relative imports when absolute_import is enabled #8858 ........ r80018 | benjamin.peterson | 2010-04-12 16:12:12 -0500 (Mon, 12 Apr 2010) | 4 lines prevent diffs from being mangled is multiprocess mode #6409 Patch by George Boutsioukis. ........ r80418 | benjamin.peterson | 2010-04-23 16:00:03 -0500 (Fri, 23 Apr 2010) | 1 line remove unhelpful description ........ r80572 | benjamin.peterson | 2010-04-27 20:33:54 -0500 (Tue, 27 Apr 2010) | 1 line use unicode literals ........ r80573 | jeffrey.yasskin | 2010-04-27 23:08:27 -0500 (Tue, 27 Apr 2010) | 6 lines Don't transform imports that are already relative. 2to3 turned from . import refactor into from .. import refactor which broke the transformation of 2to3 itself. ........ r80635 | benjamin.peterson | 2010-04-29 16:02:23 -0500 (Thu, 29 Apr 2010) | 1 line remove imports ........ r80636 | benjamin.peterson | 2010-04-29 16:02:41 -0500 (Thu, 29 Apr 2010) | 1 line unicode literal ........ r80637 | benjamin.peterson | 2010-04-29 16:03:42 -0500 (Thu, 29 Apr 2010) | 1 line must pass a string to Number ........ r80638 | benjamin.peterson | 2010-04-29 16:05:34 -0500 (Thu, 29 Apr 2010) | 1 line unicode literals ........ r80639 | benjamin.peterson | 2010-04-29 16:06:09 -0500 (Thu, 29 Apr 2010) | 1 line pass string to Number ........ r80668 | jeffrey.yasskin | 2010-04-30 18:02:47 -0500 (Fri, 30 Apr 2010) | 4 lines Make 2to3 run under Python 2.5 so that the benchmark suite at http://hg.python.org/benchmarks/ can use it and still run on implementations that haven't gotten to 2.6 yet. Fixes issue 8566. ........ r80922 | benjamin.peterson | 2010-05-07 11:06:25 -0500 (Fri, 07 May 2010) | 1 line prevent xrange transformation from wrapping range calls it produces in list ........ ................
* Merged revisions 79306,79311,79325 via svnmerge fromBenjamin Peterson2010-03-231-9/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ................ r79306 | benjamin.peterson | 2010-03-22 17:40:06 -0500 (Mon, 22 Mar 2010) | 21 lines Merged revisions 79077,79137,79304-79305 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r79077 | benjamin.peterson | 2010-03-18 18:05:29 -0500 (Thu, 18 Mar 2010) | 1 line port detect_encoding improvements from py3k ........ r79137 | benjamin.peterson | 2010-03-20 11:12:53 -0500 (Sat, 20 Mar 2010) | 1 line add a fixer for setting sys.exitfunc #2356 ........ r79304 | benjamin.peterson | 2010-03-22 17:20:22 -0500 (Mon, 22 Mar 2010) | 1 line fix test_parser when it's run in a path with spaces #7666 ........ r79305 | benjamin.peterson | 2010-03-22 17:27:07 -0500 (Mon, 22 Mar 2010) | 1 line normalize whitespace ........ ................ r79311 | benjamin.peterson | 2010-03-22 17:54:42 -0500 (Mon, 22 Mar 2010) | 9 lines Merged revisions 79309 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r79309 | benjamin.peterson | 2010-03-22 17:50:47 -0500 (Mon, 22 Mar 2010) | 1 line pass correct symbol in ........ ................ r79325 | benjamin.peterson | 2010-03-22 22:03:55 -0500 (Mon, 22 Mar 2010) | 13 lines Merged revisions 79313,79324 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r79313 | benjamin.peterson | 2010-03-22 17:59:57 -0500 (Mon, 22 Mar 2010) | 1 line another case where a symbol is needed ........ r79324 | benjamin.peterson | 2010-03-22 21:59:47 -0500 (Mon, 22 Mar 2010) | 1 line use unicode literals ........ ................
* Merged revisions 76259,76326,76376-76377,76430,76471,76517 via svnmerge fromBenjamin Peterson2009-11-251-3/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ................ r76259 | georg.brandl | 2009-11-14 05:50:51 -0600 (Sat, 14 Nov 2009) | 1 line Fix terminology. ................ r76326 | georg.brandl | 2009-11-16 10:44:05 -0600 (Mon, 16 Nov 2009) | 1 line #7302: fix link. ................ r76376 | georg.brandl | 2009-11-18 13:39:14 -0600 (Wed, 18 Nov 2009) | 1 line upcase Python ................ r76377 | georg.brandl | 2009-11-18 14:05:15 -0600 (Wed, 18 Nov 2009) | 1 line Fix markup. ................ r76430 | r.david.murray | 2009-11-20 07:29:43 -0600 (Fri, 20 Nov 2009) | 2 lines Issue 7363: fix indentation in socketserver udpserver example. ................ r76471 | georg.brandl | 2009-11-23 13:53:19 -0600 (Mon, 23 Nov 2009) | 1 line #7345: fix arguments of formatyear(). ................ r76517 | benjamin.peterson | 2009-11-25 12:16:46 -0600 (Wed, 25 Nov 2009) | 29 lines Merged revisions 76160-76161,76250,76252,76447,76506 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r76160 | benjamin.peterson | 2009-11-08 18:53:48 -0600 (Sun, 08 Nov 2009) | 1 line undeprecate the -p option; it's useful for converting python3 sources ........ r76161 | benjamin.peterson | 2009-11-08 19:05:37 -0600 (Sun, 08 Nov 2009) | 1 line simplify condition ........ r76250 | benjamin.peterson | 2009-11-13 16:56:48 -0600 (Fri, 13 Nov 2009) | 1 line fix handling of a utf-8 bom #7313 ........ r76252 | benjamin.peterson | 2009-11-13 16:58:36 -0600 (Fri, 13 Nov 2009) | 1 line remove pdb turd ........ r76447 | benjamin.peterson | 2009-11-22 18:17:40 -0600 (Sun, 22 Nov 2009) | 1 line #7375 fix nested transformations in fix_urllib ........ r76506 | benjamin.peterson | 2009-11-24 18:34:31 -0600 (Tue, 24 Nov 2009) | 1 line use generator expressions in any() ........ ................
* Merged revisions 76062 via svnmerge fromBenjamin Peterson2009-11-021-2/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ................ r76062 | benjamin.peterson | 2009-11-02 12:12:12 -0600 (Mon, 02 Nov 2009) | 70 lines Merged revisions 74359,75081,75088,75213,75278,75303,75427-75428,75734-75736,75865,76059-76061 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r74359 | benjamin.peterson | 2009-08-12 17:23:13 -0500 (Wed, 12 Aug 2009) | 1 line don't pass the deprecated print_function option ........ r75081 | benjamin.peterson | 2009-09-26 22:02:57 -0500 (Sat, 26 Sep 2009) | 1 line let 2to3 work with extended iterable unpacking ........ r75088 | benjamin.peterson | 2009-09-27 11:25:21 -0500 (Sun, 27 Sep 2009) | 1 line look on the type only for __call__ ........ r75213 | benjamin.peterson | 2009-10-03 10:09:46 -0500 (Sat, 03 Oct 2009) | 5 lines revert 75212; it's not correct People can use isinstance(x, collections.Callable) if they expect objects with __call__ in their instance dictionaries. ........ r75278 | benjamin.peterson | 2009-10-07 16:25:56 -0500 (Wed, 07 Oct 2009) | 4 lines fix whitespace problems with fix_idioms #3563 Patch by Joe Amenta. ........ r75303 | benjamin.peterson | 2009-10-09 16:59:11 -0500 (Fri, 09 Oct 2009) | 1 line port latin-1 and utf-8 cookie improvements ........ r75427 | benjamin.peterson | 2009-10-14 20:35:57 -0500 (Wed, 14 Oct 2009) | 1 line force floor division ........ r75428 | benjamin.peterson | 2009-10-14 20:39:21 -0500 (Wed, 14 Oct 2009) | 1 line silence -3 warnings about __hash__ ........ r75734 | benjamin.peterson | 2009-10-26 16:25:53 -0500 (Mon, 26 Oct 2009) | 2 lines warn on map(None, ...) with more than 2 arguments #7203 ........ r75735 | benjamin.peterson | 2009-10-26 16:28:25 -0500 (Mon, 26 Oct 2009) | 1 line remove unused result ........ r75736 | benjamin.peterson | 2009-10-26 16:29:02 -0500 (Mon, 26 Oct 2009) | 1 line using get() here is a bit pointless ........ r75865 | benjamin.peterson | 2009-10-27 15:49:00 -0500 (Tue, 27 Oct 2009) | 1 line explain reason for warning ........ r76059 | benjamin.peterson | 2009-11-02 11:43:47 -0600 (Mon, 02 Nov 2009) | 1 line tuples are no longer used for children ........ r76060 | benjamin.peterson | 2009-11-02 11:55:40 -0600 (Mon, 02 Nov 2009) | 1 line revert r76059; apparently some fixers rely on Leaf no () for children ........ r76061 | benjamin.peterson | 2009-11-02 12:06:17 -0600 (Mon, 02 Nov 2009) | 1 line make fix_tuple_params keep the tree valid #7253 ........ ................
* Merged revisions 72494 via svnmerge fromBenjamin Peterson2009-05-091-0/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ................ r72494 | benjamin.peterson | 2009-05-08 20:01:14 -0500 (Fri, 08 May 2009) | 21 lines Merged revisions 72491-72493 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r72491 | benjamin.peterson | 2009-05-08 19:33:27 -0500 (Fri, 08 May 2009) | 7 lines make 2to3 use unicode internally on 2.x This started out as a fix for #2660, but became this large refactoring when I realized the dire state this was in. 2to3 now uses tokenize.detect_encoding to decode the files correctly into unicode. ........ r72492 | benjamin.peterson | 2009-05-08 19:35:38 -0500 (Fri, 08 May 2009) | 1 line remove compat code ........ r72493 | benjamin.peterson | 2009-05-08 19:54:15 -0500 (Fri, 08 May 2009) | 1 line add a test for \r\n newlines ........ ................
* Run 2to3 on this library.Martin v. Löwis2008-03-191-7/+9
|
* Merged revisions 61596-61597 via svnmerge fromMartin v. Löwis2008-03-191-0/+405
svn+ssh://pythondev@svn.python.org/python/trunk ........ r61596 | martin.v.loewis | 2008-03-18 23:43:46 -0500 (Di, 18 Mär 2008) | 2 lines Import lib2to3. ........ r61597 | martin.v.loewis | 2008-03-18 23:58:04 -0500 (Di, 18 Mär 2008) | 3 lines Initialized merge tracking via "svnmerge" with revisions "1-61595" from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........
View Lorry Controller Status