Skip to content

buffer: add buffer.isUtf8 for utf8 validation#45947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nodejs-github-bot merged 6 commits into from
Dec 25, 2022
Merged

buffer: add buffer.isUtf8 for utf8 validation #45947

nodejs-github-bot merged 6 commits into from
Dec 25, 2022

Conversation

@anonrig
Copy link
Member

@anonriganonrig commented Dec 22, 2022•
edited
Loading

With the introduction of simdutf, we can finally have a fast way of validating utf-8 on the userland. This pull request introduces buffer.isUtf8.

This was initially requested by @KhafraDev while implementing undici websockets.

@nodejs-github-botnodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Dec 22, 2022
@anonriganonrig mentioned this pull request Dec 22, 2022
Closed
4 tasks
@anonrig
Copy link
MemberAuthor

anonrig commented Dec 22, 2022

CC @nodejs/buffer @nodejs/performance @nodejs/undici

lpinca
lpinca reviewed Dec 22, 2022
View reviewed changes
RafaelGSS
RafaelGSS reviewed Dec 22, 2022
View reviewed changes
KhafraDev
KhafraDev reviewed Dec 22, 2022
View reviewed changes
@targos
Copy link
Member

targos commented Dec 23, 2022

Can you add a test with a detached array buffer ?

lpinca
lpinca reviewed Dec 23, 2022
View reviewed changes
lpinca
lpinca reviewed Dec 23, 2022
View reviewed changes
mcollina
mcollina approved these changes Dec 23, 2022
View reviewed changes
Copy link
Member

@mcollinamcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

bnoordhuis
bnoordhuis reviewed Dec 23, 2022
View reviewed changes
@anonriganonrig requested review from KhafraDev, RafaelGSS, bnoordhuis, lpinca and targos and removed request for KhafraDevDecember 23, 2022 17:34
@anonriganonrig added request-ci Add this label to start a Jenkins CI on a PR. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. notable-change PRs with changes that should be highlighted in changelogs. labels Dec 23, 2022
lpinca
lpinca approved these changes Dec 23, 2022
View reviewed changes
@github-actionsgithub-actionsbot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 23, 2022
@github-actionsgithub-actionsbot mentioned this pull request Jan 1, 2023
Open
20 tasks
targos pushed a commit that referenced this pull request Jan 1, 2023
@anonrig@targos
buffer: add buffer.isUtf8 for utf8 validation
e445878
PR-URL: #45947 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>
@github-actionsgithub-actionsbot mentioned this pull request Jan 2, 2023
Open
22 tasks
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
01e1375
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: TBD
@RafaelGSSRafaelGSS mentioned this pull request Jan 2, 2023
Merged
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
fd77454
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
79cf6bc
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
85a26c6
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
3d55e4b
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
@github-actionsgithub-actionsbot mentioned this pull request Jan 3, 2023
Open
23 tasks
RafaelGSS added a commit that referenced this pull request Jan 3, 2023
@RafaelGSS
2023-01-04, Version 19.4.0 (Current)
ef55c9c
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig@RafaelGSS
buffer: add buffer.isUtf8 for utf8 validation
fd33718
PR-URL: #45947 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
980a9ca
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
6e0a7be
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
1ed0ef2
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
790cc5b
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig@RafaelGSS
buffer: add buffer.isUtf8 for utf8 validation
fbe399c
PR-URL: #45947 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
b181545
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
e58ed6d
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 6, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
22a2ec6
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
bentley1985
bentley1985 reviewed Jan 7, 2023
View reviewed changes
Copy link

@bentley1985bentley1985 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

juanarbol pushed a commit that referenced this pull request Jan 26, 2023
@anonrig@juanarbol
buffer: add buffer.isUtf8 for utf8 validation
907068c
PR-URL: #45947 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
de266a4
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: TBD
@juanarboljuanarbol mentioned this pull request Jan 28, 2023
Merged
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
85ed13d
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
614a60d
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 30, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
53ddd3a
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
juanarbol pushed a commit that referenced this pull request Jan 31, 2023
@anonrig@juanarbol
buffer: add buffer.isUtf8 for utf8 validation
16ee02f
PR-URL: #45947 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>
juanarbol added a commit that referenced this pull request Jan 31, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
a413280
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
@BethGriggsBethGriggs mentioned this pull request Feb 3, 2023
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lemirelemirelemire left review comments

@jasnelljasnelljasnell left review comments

@lpincalpincalpinca approved these changes

@KhafraDevKhafraDevKhafraDev left review comments

@mcollinamcollinamcollina approved these changes

@addaleaxaddaleaxaddaleax approved these changes

@ronagronagronag approved these changes

@RafaelGSSRafaelGSSRafaelGSS approved these changes

@targostargosAwaiting requested review from targos

@bnoordhuisbnoordhuisAwaiting requested review from bnoordhuis

+1 more reviewer

@bentley1985bentley1985bentley1985 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

author readyPRs that have at least one approval, no pending requests for changes, and a CI started.bufferIssues and PRs related to the buffer subsystem.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-squashAdd this label to instruct the Commit Queue to squash all the PR commits into the first one.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

13 participants

@anonrig@targos@nodejs-github-bot@mcollina@bnoordhuis@lemire@jasnell@addaleax@lpinca@ronag@RafaelGSS@KhafraDev@bentley1985