index
:
delta/cpython-git.git
2.7
3.10
3.11
3.4
3.5
3.6
3.7
3.7-readme-update
3.8
3.9
Mariatta-patch-1
Mariatta-patch-2
alex-patch-1
ast-new-type-comment-null-check
backport-0613c1e-3.7
backport-10654c1-3.7
backport-109d966-3.9
backport-24bcefc-3.6
backport-24bcefc-3.7
backport-24bcefc-3.8
backport-24bd50b-2.7
backport-24bd50b-3.6
backport-283f9a2-3.6
backport-51c3e28-3.10
backport-51c3e28-3.8
backport-52d1b86-3.6
backport-52f9842-3.9
backport-5a88d50-3.8
backport-5a8d121-3.5
backport-5bdab64-3.7
backport-6ec0dec-3.9
backport-704e2d3-2.7
backport-731ff68-3.6
backport-73d6002-3.7
backport-74ae50e-2.7
backport-74ae50e-3.7
backport-76b5d71-3.9
backport-8acd0e0-3.9
backport-8ed6503-3.8
backport-916610e-3.6
backport-916610e-3.7
backport-95da310-3.8
backport-adf24bd-3.8
backport-adf24bd-3.9
backport-cd16661-2.7
backport-eeab510-2.7
backport-eeab510-3.6
backport-eeab510-3.7
backport-f7955a8-3.10
backport-f7955a8-3.9
backport-ff023ed-3.8
benjamin-eval-loop-edit
benjamin-expat-2.2.6
benjamin-external-zip
benjamin-interp-initialize
benjamin-keyword-restriction
benjamin-musl-z-workaround
benjamin-pickle-overflow
benjamin-prlimit-permission-test
benjamin-pyapi
benjamin-tanh-zero
benjamin-traceback-obo
berkerpeksag-update-codeowners
bind-socket
bpo-37741-importlib-metadata-docs-visible
bpo-39107
bpo-39235-genexp-endloc
bpo-41855-backport-fix
bpo-42057
bpo-43428/importlib-metadata-3.7
bpo-43569/man-as-machine
bpo-43780/importlib_metadata-3.10
bpo-43856/importlib-metadata-new
bpo-43926/importlib-metadata-4
bpo-45419
bpo-45514/deprecate-importlib-resources-legacy
bpo-45765/metadata-distribution-empty-path
bpo-46118/resources-package
bpo-46124/zoneinfo-resources
bpo-46125/importlib-resources-test-traversable
bpo-46126/bad-error-message
bpo-46425/restore-simple-importlib-tests
bpo-46474/entrypoint-regex
bpo-46474/entrypoint-regex-fix
branch-v3.10.0
branch-v3.9.0
bugfix/16396-linux-wintypes-importable
buildbot-custom
cjw296-patch-1
constant-kind
datetime_backport_test
dependabot/github_actions/actions/cache-2.1.7
dependabot/github_actions/actions/cache-v2.1.3
dependabot/github_actions/actions/cache-v2.1.4
dependabot/github_actions/actions/cache-v2.1.5
dependabot/github_actions/actions/upload-artifact-2.3.1
dependabot/github_actions/actions/upload-artifact-v2.2.1
dependabot/github_actions/actions/upload-artifact-v2.2.2
dependabot/github_actions/actions/upload-artifact-v2.2.3
descr-howto-feedback
enum-lost-fixes
enum-private-310
ethanfurman-acks
faster-bytes-iter
feature-version
feature/42129-resources-namespace-packages
feature/putrequest-hooks
fedora-2.7
find_module-deprecation-windows
fix-cf-feature-versiob
fix-cf-feature-version
fix-cf-feature-version-38
fix-cf-feature-version-39
fix-misc-acks
fix-misc-acks-38
fix-misc-acks-sort
fix-namedexpr-comment
fix-req-type
fix-segfaults
fix-time_ns-link
fix-tok-type-comments-flag
fix-traceback-syntax-error
fixup-bpo-34003
importlib_metadata-3.3
main
master
misc-acks-comment
mp-manager-tests
ncoghlan-bpo-33499-whats-new
ncoghlan-clarify-U-mode-deprecation
ncoghlan-fix-versionchanged-note
ncoghlan-whatsnew-38-placeholder-notes
orphaned-comment
pep572
pipelines-badge-3.6
pipelines-readme-3.7
pr_18370
pr_27789
pr_4170
print-statement
readme-update
refactor-lambda-parameters
revert-10245-msix
revert-10341-patch-7
revert-11293-bpo35565
revert-11318-fast-enum
revert-11341-bpo35603
revert-11477-backport-cbb1645-3.6
revert-11482-backport-cbb1645-2.7
revert-13569-undo-ea2b76bdc5f97f49701213d105b8ec2387ea2fa5
revert-13599-codeowner
revert-13893-fix-issue-37193
revert-14522-remove_unused_imports37
revert-15687-backport-87bd207-3.7
revert-15895-named_tuple_concept
revert-16839-backport-5a88d50-3.8
revert-17659-utf8-buffer
revert-17774-issue27973-win-fix
revert-19108-remove-cElementTree
revert-22049-bpo-1635741-signalmodule
revert-22513-update_test_codecs
revert-22997-tempfile-overhead
revert-23107-revert-13893-fix-issue-37193
revert-23398-bpo-1635741-struct
revert-23827-backport-dcc997c-3.9
revert-23828-backport-dcc997c-3.8
revert-23917-enum-prepare_kwds
revert-24821-fix/bigint_freelist
revert-25103-binary-locale
revert-25134-regen_stable_abi
revert-28650-revert-28615-test-datetime-discover
revert-28660-revert-gh28645
revert-29124-bpo-44019
revert-30472-issue-46301
revert-6121-is_integer
revert-7960-backport-2cc9d21-3.7
revert-9703-master
soft-keywords
suspcious
sync-39-whatsnew
travis-3.5
travis-3.6
travis-macOS
type-comments
unixdeepfreeze
v3.8.4
whatsnew-617
windeepfreeze
zhangyangyu-patch-1
zooba-patch-1
github.com: python/cpython.git
summary
refs
log
tree
commit
diff
log msg
author
committer
range
path:
root
/
Lib
/
test
/
test_tokenize.py
Commit message (
Expand
)
Author
Age
Files
Lines
*
bpo-35224: PEP 572 Implementation (#10497)
Emily Morehouse
2019-01-24
1
-0
/
+1
*
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-...
Serhiy Storchaka
2018-12-22
1
-0
/
+2
*
Fixed several assertTrue() that were intended to be assertEqual(). (GH-8191)
Sergey Fedoseev
2018-07-09
1
-1
/
+1
*
bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (G...
Ammar Askar
2018-07-06
1
-24
/
+47
*
bpo-12486: Document tokenize.generate_tokens() as public API (#6957)
Thomas Kluyver
2018-06-05
1
-2
/
+15
*
bpo-30406: Make async and await proper keywords (#1669)
Jelle Zijlstra
2017-10-05
1
-14
/
+14
*
bpo-31029: test_tokenize Add missing import unittest (#2865)
Stéphane Wirtel
2017-07-25
1
-0
/
+1
*
bpo-25324: copy tok_name before changing it (#1608)
Albert-Jan Nijburg
2017-05-31
1
-6
/
+6
*
bpo-30377: Simplify handling of COMMENT and NL in tokenize.py (#1607)
Albert-Jan Nijburg
2017-05-24
1
-7
/
+9
*
Add ELLIPSIS and RARROW. Add tests (#666)
Jim Fasarakis-Hilliard
2017-03-14
1
-0
/
+21
*
Issue #26331: Implement the parsing part of PEP 515.
Brett Cannon
2016-09-09
1
-7
/
+23
*
Rename test_pep####.py files
Zachary Ware
2016-09-09
1
-5
/
+6
*
Fix running test_tokenize directly
Zachary Ware
2016-09-09
1
-2
/
+2
*
Issue 25311: Add support for f-strings to tokenize.py. Also added some commen...
Eric V. Smith
2015-10-26
1
-0
/
+17
*
Issue 25422: Add tests for multi-line string tokenization. Also remove trunca...
Eric V. Smith
2015-10-16
1
-6
/
+32
*
Issue #25317: Converted doctests in test_tokenize to unittests.
Serhiy Storchaka
2015-10-06
1
-419
/
+398
|
\
|
*
Issue #25317: Converted doctests in test_tokenize to unittests.
Serhiy Storchaka
2015-10-06
1
-357
/
+332
*
|
Issue #24619: Simplify async/await tokenization.
Yury Selivanov
2015-07-23
1
-0
/
+73
*
|
Issue #24619: New approach for tokenizing async/await.
Yury Selivanov
2015-07-22
1
-2
/
+13
*
|
Issue #20387: Merge test and patch from 3.4.4
Jason R. Coombs
2015-06-28
1
-1
/
+20
|
\
\
|
|
/
|
*
Issue #20387: Correct test to properly capture expectation.
Jason R. Coombs
2015-06-25
1
-2
/
+2
|
*
Issue #20387: Add test capturing failure to roundtrip indented code in tokeni...
Jason R. Coombs
2015-06-20
1
-0
/
+17
|
*
Remove unused import and remove doctest-only import into doctests.
Jason R. Coombs
2015-06-20
1
-1
/
+3
*
|
(Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary file
Victor Stinner
2015-05-26
1
-1
/
+9
|
\
\
|
|
/
|
*
Issue #23840: tokenize.open() now closes the temporary binary file on error to
Victor Stinner
2015-05-26
1
-1
/
+9
*
|
Issue 24226: Fix parsing of many sequential one-line 'def' statements.
Yury Selivanov
2015-05-18
1
-0
/
+11
*
|
PEP 0492 -- Coroutines with async and await syntax. Issue #24017.
Yury Selivanov
2015-05-11
1
-0
/
+186
*
|
Issue #23681: Fixed Python 2 to 3 poring bugs.
Serhiy Storchaka
2015-03-20
1
-3
/
+4
|
\
\
|
|
/
|
*
Issue #23681: Fixed Python 2 to 3 poring bugs.
Serhiy Storchaka
2015-03-20
1
-3
/
+4
*
|
PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)
Benjamin Peterson
2014-04-09
1
-1
/
+4
|
/
*
Issue #9974: When untokenizing, use row info to insert backslash+newline.
Terry Jan Reedy
2014-02-23
1
-1
/
+16
*
Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The
Terry Jan Reedy
2014-02-23
1
-14
/
+38
*
Issue #8478: Untokenizer.compat now processes first token from iterator input.
Terry Jan Reedy
2014-02-17
1
-0
/
+13
*
whitespace
Terry Jan Reedy
2014-02-17
1
-2
/
+2
*
Untokenize: An logically incorrect assert tested user input validity.
Terry Jan Reedy
2014-02-17
1
-1
/
+15
*
Issue #18960: Fix bugs with Python source code encoding in the second line.
Serhiy Storchaka
2014-01-09
1
-0
/
+33
*
Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
Serhiy Storchaka
2013-09-16
1
-0
/
+7
*
#16152: merge with 3.2.
Ezio Melotti
2012-11-03
1
-0
/
+4
|
\
|
*
#16152: fix tokenize to ignore whitespace at the end of the code when no newl...
Ezio Melotti
2012-11-03
1
-0
/
+5
*
|
Merge branch
Florent Xicluna
2012-07-07
1
-0
/
+4
|
\
\
|
|
/
|
*
Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding d...
Florent Xicluna
2012-07-07
1
-0
/
+4
*
|
Issue #15096: Drop support for the ur string prefix
Christian Heimes
2012-06-20
1
-20
/
+2
*
|
Issue #15054: Fix incorrect tokenization of 'b' string literals.
Meador Inge
2012-06-16
1
-0
/
+76
*
|
Issue #14629: Mention the filename in SyntaxError exceptions from
Brett Cannon
2012-04-20
1
-0
/
+29
*
|
merge 3.2: issue 14629
Martin v. Löwis
2012-04-20
1
-0
/
+10
|
\
\
|
|
/
|
*
Issue #14629: Raise SyntaxError in tokenizer.detect_encoding
Martin v. Löwis
2012-04-20
1
-0
/
+10
*
|
Updated tokenize to support the inverse byte literals new in 3.3
Armin Ronacher
2012-03-04
1
-0
/
+12
*
|
Issue #2134: Add support for tokenize.TokenInfo.exact_type.
Meador Inge
2012-01-19
1
-1
/
+74
*
|
#13012: use splitlines(keepends=True/False) instead of splitlines(0/1).
Ezio Melotti
2011-09-28
1
-1
/
+1
|
/
*
tokenize is just broken on test_pep3131.py
Benjamin Peterson
2011-08-13
1
-0
/
+3
[next]