gh-135676: Add a summary of source characters

[encukou]

The lexical analysis docs have notes like this at the end:

• The period can also occur in floating-point and imaginary literals.

• The following printing ASCII characters have special meaning as part of other tokens or are otherwise significant to the lexical analyzer: ' " # \

• The following printing ASCII characters are not used in Python. Their occurrence outside string literals and comments is an unconditional error: $ ? `

The intent behind these seems to be providing a "map" of what all the ASCII characters do in Python, but that map is incomplete as it is, and isn't really kept up to date.

This instead provides a summary of source characters -- nominally the ones that start tokens, with notes for other notable cases.
The table can also serve as an alternate "table of contents".

The presentation -- a table of bulleted lists -- is a bit wacky but I think it gets the job done.

• Issue: Reword the Lexical Analysis chapter of the docs #135676

---

📚 Documentation preview 📚: https://cpython-previews--138194.org.readthedocs.build/

[AA-Turner]

I think this is a useful addition!

A

[AA-Turner]

I'm not sure this note is needed?

[willingc]

I agree with @AA-Turner. Stream and sequence are both overloaded terms that may be better unpacked by the reader in context.

[encukou]

OK; I've removed it

[AA-Turner]

In general for list tables it can be useful to alternate list markers, e.g. using - to denote items of the second-level list. Not essential, though.

[encukou]

All my list-tables will do that from now on :)

[AA-Turner]

Is 'non-ASCII character' too broad here? Not all characters can form valid identifiers, especially if expanding to the full Unicode space!

[encukou]

It is broad, but: if the tokenizer sees a non-ASCII character, the next token can only be a NAME (or error). (Except inside strings/comments, but then it's not deciding what the next token will be.)

If I remember correctly¹, the tokenizer implementation does lump non-ASCII characters with the letters, and only checks validity after it parses an identifier-like token.

¹ Maybe I don't, but it certainly could do that :)

[willingc]

A nice improvement @encukou. I've left a few prose suggestions but fine as is too. Thanks!

[willingc]

I agree with @AA-Turner. Stream and sequence are both overloaded terms that may be better unpacked by the reader in context.

[miss-islington-app]

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

[bedevere-app]

GH-139781 is a backport of this pull request to the 3.14 branch.

[encukou]

Thank you for the reviews!