index
:
delta/cpython-git.git
2.7
3.10
3.11
3.4
3.5
3.6
3.7
3.7-readme-update
3.8
3.9
Mariatta-patch-1
Mariatta-patch-2
alex-patch-1
ast-new-type-comment-null-check
backport-0613c1e-3.7
backport-10654c1-3.7
backport-109d966-3.9
backport-24bcefc-3.6
backport-24bcefc-3.7
backport-24bcefc-3.8
backport-24bd50b-2.7
backport-24bd50b-3.6
backport-283f9a2-3.6
backport-51c3e28-3.10
backport-51c3e28-3.8
backport-52d1b86-3.6
backport-52f9842-3.9
backport-5a88d50-3.8
backport-5a8d121-3.5
backport-5bdab64-3.7
backport-6ec0dec-3.9
backport-704e2d3-2.7
backport-731ff68-3.6
backport-73d6002-3.7
backport-74ae50e-2.7
backport-74ae50e-3.7
backport-76b5d71-3.9
backport-8acd0e0-3.9
backport-8ed6503-3.8
backport-916610e-3.6
backport-916610e-3.7
backport-95da310-3.8
backport-adf24bd-3.8
backport-adf24bd-3.9
backport-cd16661-2.7
backport-eeab510-2.7
backport-eeab510-3.6
backport-eeab510-3.7
backport-f7955a8-3.10
backport-f7955a8-3.9
backport-ff023ed-3.8
benjamin-eval-loop-edit
benjamin-expat-2.2.6
benjamin-external-zip
benjamin-interp-initialize
benjamin-keyword-restriction
benjamin-musl-z-workaround
benjamin-pickle-overflow
benjamin-prlimit-permission-test
benjamin-pyapi
benjamin-tanh-zero
benjamin-traceback-obo
berkerpeksag-update-codeowners
bind-socket
bpo-37741-importlib-metadata-docs-visible
bpo-39107
bpo-39235-genexp-endloc
bpo-41855-backport-fix
bpo-42057
bpo-43428/importlib-metadata-3.7
bpo-43569/man-as-machine
bpo-43780/importlib_metadata-3.10
bpo-43856/importlib-metadata-new
bpo-43926/importlib-metadata-4
bpo-45419
bpo-45514/deprecate-importlib-resources-legacy
bpo-45765/metadata-distribution-empty-path
bpo-46118/resources-package
bpo-46124/zoneinfo-resources
bpo-46125/importlib-resources-test-traversable
bpo-46126/bad-error-message
bpo-46425/restore-simple-importlib-tests
bpo-46474/entrypoint-regex
bpo-46474/entrypoint-regex-fix
branch-v3.10.0
branch-v3.9.0
bugfix/16396-linux-wintypes-importable
buildbot-custom
cjw296-patch-1
constant-kind
datetime_backport_test
dependabot/github_actions/actions/cache-2.1.7
dependabot/github_actions/actions/cache-v2.1.3
dependabot/github_actions/actions/cache-v2.1.4
dependabot/github_actions/actions/cache-v2.1.5
dependabot/github_actions/actions/upload-artifact-2.3.1
dependabot/github_actions/actions/upload-artifact-v2.2.1
dependabot/github_actions/actions/upload-artifact-v2.2.2
dependabot/github_actions/actions/upload-artifact-v2.2.3
descr-howto-feedback
enum-lost-fixes
enum-private-310
ethanfurman-acks
faster-bytes-iter
feature-version
feature/42129-resources-namespace-packages
feature/putrequest-hooks
fedora-2.7
find_module-deprecation-windows
fix-cf-feature-versiob
fix-cf-feature-version
fix-cf-feature-version-38
fix-cf-feature-version-39
fix-misc-acks
fix-misc-acks-38
fix-misc-acks-sort
fix-namedexpr-comment
fix-req-type
fix-segfaults
fix-time_ns-link
fix-tok-type-comments-flag
fix-traceback-syntax-error
fixup-bpo-34003
importlib_metadata-3.3
main
master
misc-acks-comment
mp-manager-tests
ncoghlan-bpo-33499-whats-new
ncoghlan-clarify-U-mode-deprecation
ncoghlan-fix-versionchanged-note
ncoghlan-whatsnew-38-placeholder-notes
orphaned-comment
pep572
pipelines-badge-3.6
pipelines-readme-3.7
pr_18370
pr_27789
pr_4170
print-statement
readme-update
refactor-lambda-parameters
revert-10245-msix
revert-10341-patch-7
revert-11293-bpo35565
revert-11318-fast-enum
revert-11341-bpo35603
revert-11477-backport-cbb1645-3.6
revert-11482-backport-cbb1645-2.7
revert-13569-undo-ea2b76bdc5f97f49701213d105b8ec2387ea2fa5
revert-13599-codeowner
revert-13893-fix-issue-37193
revert-14522-remove_unused_imports37
revert-15687-backport-87bd207-3.7
revert-15895-named_tuple_concept
revert-16839-backport-5a88d50-3.8
revert-17659-utf8-buffer
revert-17774-issue27973-win-fix
revert-19108-remove-cElementTree
revert-22049-bpo-1635741-signalmodule
revert-22513-update_test_codecs
revert-22997-tempfile-overhead
revert-23107-revert-13893-fix-issue-37193
revert-23398-bpo-1635741-struct
revert-23827-backport-dcc997c-3.9
revert-23828-backport-dcc997c-3.8
revert-23917-enum-prepare_kwds
revert-24821-fix/bigint_freelist
revert-25103-binary-locale
revert-25134-regen_stable_abi
revert-28650-revert-28615-test-datetime-discover
revert-28660-revert-gh28645
revert-29124-bpo-44019
revert-30472-issue-46301
revert-6121-is_integer
revert-7960-backport-2cc9d21-3.7
revert-9703-master
soft-keywords
suspcious
sync-39-whatsnew
travis-3.5
travis-3.6
travis-macOS
type-comments
unixdeepfreeze
v3.8.4
whatsnew-617
windeepfreeze
zhangyangyu-patch-1
zooba-patch-1
github.com: python/cpython.git
summary
refs
log
tree
commit
diff
log msg
author
committer
range
path:
root
/
Lib
/
tokenize.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
bpo-33899: Revert tokenize module adding an implicit final NEWLINE (GH-10072)
Tal Einat
2018-10-24
1
-10
/
+0
*
[2.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behav...
Ammar Askar
2018-07-06
1
-0
/
+10
*
Issue #20387: Backport fix from Python 3.4
Jason R. Coombs
2015-06-28
1
-0
/
+17
*
Issue #9974: When untokenizing, use row info to insert backslash+newline.
Terry Jan Reedy
2014-02-23
1
-0
/
+6
*
whitespace
Terry Jan Reedy
2014-02-17
1
-1
/
+1
*
Issue #8478: Untokenizer.compat now processes first token from iterator input.
Terry Jan Reedy
2014-02-17
1
-9
/
+7
*
Untokenize: An logically incorrect assert tested user input validity.
Terry Jan Reedy
2014-02-17
1
-1
/
+3
*
#19620: Fix typo in docstring (noticed by Christopher Welborn).
Ezio Melotti
2013-11-25
1
-1
/
+1
*
#16152: fix tokenize to ignore whitespace at the end of the code when no newl...
Ezio Melotti
2012-11-03
1
-1
/
+3
*
Issue #15054: Fix incorrect tokenization of 'b' and 'br' string literals.
Meador Inge
2012-06-16
1
-5
/
+5
*
some cleanups
Benjamin Peterson
2009-10-15
1
-10
/
+10
*
use floor division and add a test that exercises the tabsize codepath
Benjamin Peterson
2009-10-15
1
-1
/
+1
*
pep8ify if blocks
Benjamin Peterson
2009-10-15
1
-9
/
+18
*
Remove a tuple unpacking in a parameter list to remove a SyntaxWarning raised
Brett Cannon
2008-08-02
1
-1
/
+3
*
revert 63965 for preformance reasons
Benjamin Peterson
2008-06-05
1
-1
/
+1
*
use the more idomatic while True
Benjamin Peterson
2008-06-05
1
-1
/
+1
*
Issue2495: tokenize.untokenize did not insert space between two consecutive s...
Amaury Forgeot d'Arc
2008-03-27
1
-1
/
+10
*
Added PEP 3127 support to tokenize (with tests); added PEP 3127 to NEWS.
Eric Smith
2008-03-17
1
-2
/
+3
*
Fix #1679: "0x" was taken as a valid integer literal.
Georg Brandl
2008-01-19
1
-1
/
+1
*
Added bytes and b'' as aliases for str and ''
Christian Heimes
2008-01-18
1
-3
/
+16
*
Add name to credits (for untokenize).
Raymond Hettinger
2006-12-02
1
-1
/
+1
*
Replace dead code with an assert.
Jeremy Hylton
2006-08-23
1
-4
/
+1
*
Bug fixes large and small for tokenize.
Jeremy Hylton
2006-08-23
1
-31
/
+79
*
Make tabnanny recognize IndentationErrors raised by tokenize.
Georg Brandl
2006-08-14
1
-1
/
+2
*
Insert a safety space after numbers as well as names in untokenize().
Guido van Rossum
2006-03-30
1
-1
/
+1
*
SF bug #1224621: tokenize module does not detect inconsistent dedents
Raymond Hettinger
2005-06-21
1
-0
/
+3
*
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger
2005-06-10
1
-3
/
+49
*
PEP-0318, @decorator-style. In Guido's words:
Anthony Baxter
2004-08-02
1
-1
/
+1
*
Get rid of many apply() calls.
Guido van Rossum
2003-02-27
1
-3
/
+3
*
SF 633560: tokenize.__all__ needs "generate_tokens"
Raymond Hettinger
2002-11-05
1
-1
/
+2
*
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum
2002-08-24
1
-10
/
+19
*
Whitespace normalization.
Tim Peters
2002-05-23
1
-5
/
+5
*
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger
2002-05-15
1
-0
/
+25
*
Remove some now-obsolete generator future statements.
Tim Peters
2002-04-01
1
-2
/
+0
*
Cleanup x so it is not left in module
Neal Norwitz
2002-03-26
1
-0
/
+1
*
SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters
2001-08-30
1
-2
/
+2
*
Add new tokens // and //=, in support of PEP 238.
Guido van Rossum
2001-08-08
1
-0
/
+1
*
Use string.ascii_letters instead of string.letters (SF bug #226706).
Fred Drake
2001-07-20
1
-1
/
+1
*
Preliminary support for "from __future__ import generators" to enable
Guido van Rossum
2001-07-15
1
-0
/
+2
*
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters
2001-06-29
1
-8
/
+21
*
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters
2001-06-18
1
-15
/
+20
*
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee
2001-03-23
1
-10
/
+11
*
Better __credits__.
Ka-Ping Yee
2001-03-01
1
-1
/
+2
*
Add __author__ and __credits__ variables.
Ka-Ping Yee
2001-03-01
1
-1
/
+2
*
final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be
Skip Montanaro
2001-03-01
1
-1
/
+5
*
String method conversion.
Eric S. Raymond
2001-02-09
1
-1
/
+1
*
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee
2001-01-15
1
-9
/
+25
*
Whitespace normalization.
Tim Peters
2001-01-15
1
-1
/
+0
*
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters
2000-10-07
1
-12
/
+20
*
Update for augmented assignment, tested & approved by Guido.
Thomas Wouters
2000-08-24
1
-2
/
+5
[next]