Tokenizer module does not handle backslash characters correctly

[ucodery]

BPO	46274
Nosy	@lysnikolaou, @ucodery

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-01-05.23:12:47.391>
labels = ['interpreter-core', '3.8', '3.9', '3.10', '3.11', '3.7']
title = 'Tokenizer module does not handle backslash characters correctly'
updated_at = <Date 2022-01-06.13:57:33.135>
user = 'https://github.com/ucodery'

bugs.python.org fields:

activity = <Date 2022-01-06.13:57:33.135>
actor = 'pablogsal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Parser']
creation = <Date 2022-01-05.23:12:47.391>
creator = 'ucodery'
dependencies = []
files = []
hgrepos = []
issue_num = 46274
keywords = []
message_count = 1.0
messages = ['409811']
nosy_count = 2.0
nosy_names = ['lys.nikolaou', 'ucodery']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46274'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[ucodery]

A source of one or more backslash-escaped newlines, and one final newline, is not tokenized the same as a source where those lines are "manually joined".

The source

\
\
\

produces the tokens NEWLINE, ENDMARKER when piped to the tokenize module.

Whereas the source

produces the tokens NL, ENDMARKER.

What I expect is to receive only one NL token from both sources. As per the documentation "Two or more physical lines may be joined into logical lines using backslash characters" ... "A logical line that contains only spaces, tabs, formfeeds and possibly a comment, is ignored (i.e., no NEWLINE token is generated)"

And, because these logical lines are not being ignored, if there are spaces/tabs, INDENT and DEDENT tokens are also being unexpectedly produced.

The source

 \

produces the tokens INDENT, NEWLINE, DEDENT, ENDMARKER.

Whereas the source (with spaces)

produces the tokens NL, ENDMARKER.