gh-105017: Include CRLF lines in strings and column numbers

[mgmacias95]

• Issue: Tokenizer produces different output on Windows on py312 for ends of files #105017

[pablogsal]

There were a bunch of failing tests because we are removing CRLF handling in the regular path that expects the source to be normalized. I pushed a commit that only preserves CRLF in our tokenize extension

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 363de28 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit d3d4ff5 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[miss-islington]

Thanks @mgmacias95 for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[bedevere-bot]

GH-105041 is a backport of this pull request to the 3.12 branch.

[pablogsal]

LGTM

Great job! ✨

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.12 has failed when building commit 3f8d5d9.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1102/builds/33) and take a look at the build logs.
4. Check if the failure is related to this commit (3f8d5d9) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1102/builds/33

Failed tests:

• test_threading
• test__xxsubinterpreters

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 16, done. remote: Counting objects: 6% (1/16) remote: Counting objects: 12% (2/16) remote: Counting objects: 18% (3/16) remote: Counting objects: 25% (4/16) remote: Counting objects: 31% (5/16) remote: Counting objects: 37% (6/16) remote: Counting objects: 43% (7/16) remote: Counting objects: 50% (8/16) remote: Counting objects: 56% (9/16) remote: Counting objects: 62% (10/16) remote: Counting objects: 68% (11/16) remote: Counting objects: 75% (12/16) remote: Counting objects: 81% (13/16) remote: Counting objects: 87% (14/16) remote: Counting objects: 93% (15/16) remote: Counting objects: 100% (16/16) remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 6% (1/15) remote: Compressing objects: 13% (2/15) remote: Compressing objects: 20% (3/15) remote: Compressing objects: 26% (4/15) remote: Compressing objects: 33% (5/15) remote: Compressing objects: 40% (6/15) remote: Compressing objects: 46% (7/15) remote: Compressing objects: 53% (8/15) remote: Compressing objects: 60% (9/15) remote: Compressing objects: 66% (10/15) remote: Compressing objects: 73% (11/15) remote: Compressing objects: 80% (12/15) remote: Compressing objects: 86% (13/15) remote: Compressing objects: 93% (14/15) remote: Compressing objects: 100% (15/15) remote: Compressing objects: 100% (15/15), done. remote: Total 16 (delta 1), reused 11 (delta 1), pack-reused 0  From https://github.com/python/cpython * branch 3.12 -> FETCH_HEAD Note: switching to '3f8d5d9ed60ae0f70c92dc29587cc6367eda80c3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name> Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 3f8d5d9ed6 [3.12] gh-105017: Include CRLF lines in strings and column numbers (GH-105030) (#105041) Switched to and reset branch '3.12' configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly. Fatal Python error: Segmentation fault Thread 0x00007f253095a280 (most recent call first): File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 74 in _write_status File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 95 in addSuccess File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/testresult.py", line 122 in addSuccess File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 648 in run File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 690 in __call__ File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__ File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__ File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__ File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 240 in run File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1115 in _run_suite File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1241 in run_unittest File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2 File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 278 in runtest File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 783 in _main File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 738 in main File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 802 in main File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/__main__.py", line 2 in <module> File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main Extension modules: _testcapi (total: 1) Segmentation fault make: *** [Makefile:2009: buildbottest] Error 139 Cannot open file '/root/buildarea/3.12.angelico-debian-amd64/build/test-results.xml' for upload