Skip to content

gh-94823: Improve coverage in tokenizer.c:valid_utf8#94856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
miss-islington merged 4 commits into from
Aug 16, 2022
Merged

gh-94823: Improve coverage in tokenizer.c:valid_utf8 #94856

miss-islington merged 4 commits into from
Aug 16, 2022

Conversation

@mdboom
Copy link
Contributor

@mdboommdboom commented Jul 14, 2022•
edited by miss-islington
Loading

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in unicode_decode_utf8. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.

Automerge-Triggered-By: GH:ericsnowcurrently

@bedevere-botbedevere-bot added tests Tests in the Lib/test dir awaiting review labels Jul 14, 2022
@brandtbucherbrandtbucher added needs backport to 3.11 only security fixes labels Jul 15, 2022
@mdboom
pythongh-94823: Improve coverage in tokenizer.c:valid_utf8
e52f328
When loading a source file from disk, there is a separate UTF-8 validator distinct from the one in `unicode_decode_utf8`. This exercises that code path with the same set of invalid inputs as we use for testing the "other" UTF-8 decoder.
@mdboommdboomforce-pushed the tokenizer-valid-utf8-coverage branch from 6db7d84 to e52f328CompareJuly 15, 2022 18:11
ericsnowcurrently
ericsnowcurrently requested changes Aug 12, 2022
View reviewed changes
Copy link
Member

@ericsnowcurrentlyericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! It mostly looks good. I noticed a couple of small typos.

@bedevere-bot
Copy link

bedevere-bot commented Aug 12, 2022

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@mdboom
Copy link
ContributorAuthor

mdboom commented Aug 16, 2022

I have made the requested changes; please review again

@bedevere-bot
Copy link

bedevere-bot commented Aug 16, 2022

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

ericsnowcurrently
ericsnowcurrently approved these changes Aug 16, 2022
View reviewed changes
Copy link
Member

@ericsnowcurrentlyericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@miss-islingtonmiss-islington merged commit f215d7c into python:mainAug 16, 2022
@miss-islington
Copy link
Contributor

miss-islington commented Aug 16, 2022

Thanks @mdboom for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 16, 2022
@mdboom@miss-islington
pythongh-94823: Improve coverage in tokenizer.c:valid_utf8 (pythonGH-…
0425b23
…94856) When loading a source file from disk, there is a separate UTF-8 validator distinct from the one in `unicode_decode_utf8`. This exercises that code path with the same set of invalid inputs as we use for testing the "other" UTF-8 decoder. (cherry picked from commit f215d7c) Co-authored-by: Michael Droettboom <[email protected]>
@bedevere-botbedevere-bot removed the needs backport to 3.11 only security fixes label Aug 16, 2022
@bedevere-bot
Copy link

bedevere-bot commented Aug 16, 2022

GH-96029 is a backport of this pull request to the 3.11 branch.

@ericsnowcurrently
Copy link
Member

ericsnowcurrently commented Aug 16, 2022

Thanks for the test!

pablogsal pushed a commit that referenced this pull request Aug 16, 2022
@miss-islington@mdboom
[3.11]gh-94823: Improve coverage in tokenizer.c:valid_utf8 (GH-94856) (
2bb363c
#96029) Co-authored-by: Michael Droettboom <[email protected]>
This was referenced Aug 22, 2022
Closed
Closed
Merged
@kumaraditya303kumaraditya303 mentioned this pull request Sep 17, 2022
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrentlyericsnowcurrentlyericsnowcurrently approved these changes

Assignees

No one assigned

Labels

skip newstestsTests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@mdboom@bedevere-bot@miss-islington@ericsnowcurrently@brandtbucher@kumaraditya303