[v18.x backport] test_runner features#46839

MoLow · 2023-02-25T21:26:42Z

since these commits depend on each other and many of them don't land clearly - backporting these in bulk makes more sense

Fixes: nodejs#44612 PR-URL: nodejs#45264 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: nodejs#45712Fixes: nodejs#45648 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

Move the logic for handling --test-reporter out of the general module loader and into the test_runner subsystem. PR-URL: nodejs#45923 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#46030 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

This commit updates the test runner to make the built in test reporters internal modules. PR-URL: nodejs#46092 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

nodejs-github-bot · 2023-02-25T21:26:46Z

Review requested:

@nodejs/test_runner

This commit adds code coverage functionality to the node:test module. When node:test is used in conjunction with the new --test-coverage CLI flag, a coverage report is created when the test runner finishes. The coverage summary is forwarded to any test runner reporters so that the display can be customized as desired. This new functionality is compatible with the existing NODE_V8_COVERAGE environment variable as well. There are still several limitations, which will be addressed in subsequent pull requests: - Coverage is only reported for a single process. It is possible to merge coverage reports together. Once this is done, the --test flag will be supported as well. - Source maps are not currently supported. - Excluding specific files or directories from the coverage report is not currently supported. Node core modules and node_modules/ are excluded though. PR-URL: nodejs#46017 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

Add experimental to the name as requested during review. PR-URL: nodejs#46017 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

PR-URL: nodejs#46311Fixes: nodejs#45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: nodejs#46441Fixes: nodejs#45910 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#46450Fixes: nodejs#45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#46056Fixes: nodejs#46048 Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: nodejs#46544Fixes: nodejs#46478 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs#46440Fixes: nodejs#45833 Refs: nodejs#45833 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: nodejs#45736 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: nodejs#46651 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

The test runner relies on a few CLI options. That code was spread across a few locations. This commit centralizes that logic. PR-URL: nodejs#46707 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs#46540Fixes: nodejs#46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs#46737Fixes: nodejs#46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

PR-URL: nodejs#46604Fixes: nodejs#46562 Reviewed-By: Moshe Atlow <[email protected]>

The test runner is bootstrapped synchronously, with the exception of importing custom reporters. To better handle asynchronously throw errors, this commit introduces an internal error type that can be checked for from the test runner's uncaughtException handler. After nodejs#46707 and this change land, the other throw statement in the uncaughtException handler can be removed. This will allow the test runner to handle errors thrown from outside of tests (which currently prevents the test runner from reporting results). PR-URL: nodejs#46720 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

The test runner relies on a few CLI options. That code was spread across a few locations. This commit centralizes that logic. PR-URL: #46707 Backport-PR-URL: #46839 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46540 Backport-PR-URL: #46839Fixes: #46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46737 Backport-PR-URL: #46839Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

PR-URL: #46604 Backport-PR-URL: #46839Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

The test runner is bootstrapped synchronously, with the exception of importing custom reporters. To better handle asynchronously throw errors, this commit introduces an internal error type that can be checked for from the test runner's uncaughtException handler. After #46707 and this change land, the other throw statement in the uncaughtException handler can be removed. This will allow the test runner to handle errors thrown from outside of tests (which currently prevents the test runner from reporting results). PR-URL: #46720 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

juanarbol · 2023-03-03T06:50:29Z

Landed in 9434396...e17d857 🎉

Fixes: #44612 PR-URL: #45264 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #45712 Backport-PR-URL: #46839Fixes: #45648 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

Move the logic for handling --test-reporter out of the general module loader and into the test_runner subsystem. PR-URL: #45923 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #46030 Backport-PR-URL: #46839 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

This commit updates the test runner to make the built in test reporters internal modules. PR-URL: #46092 Backport-PR-URL: #46839 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

This commit adds code coverage functionality to the node:test module. When node:test is used in conjunction with the new --test-coverage CLI flag, a coverage report is created when the test runner finishes. The coverage summary is forwarded to any test runner reporters so that the display can be customized as desired. This new functionality is compatible with the existing NODE_V8_COVERAGE environment variable as well. There are still several limitations, which will be addressed in subsequent pull requests: - Coverage is only reported for a single process. It is possible to merge coverage reports together. Once this is done, the --test flag will be supported as well. - Source maps are not currently supported. - Excluding specific files or directories from the coverage report is not currently supported. Node core modules and node_modules/ are excluded though. PR-URL: #46017 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

Add experimental to the name as requested during review. PR-URL: #46017 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

PR-URL: #46311 Backport-PR-URL: #46839Fixes: #45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #46441 Backport-PR-URL: #46839Fixes: #45910 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #46450 Backport-PR-URL: #46839Fixes: #45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #46056 Backport-PR-URL: #46839Fixes: #46048 Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: #46544 Backport-PR-URL: #46839Fixes: #46478 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46440 Backport-PR-URL: #46839Fixes: #45833 Refs: #45833 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

The test runner relies on a few CLI options. That code was spread across a few locations. This commit centralizes that logic. PR-URL: #46707 Backport-PR-URL: #46839 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46540 Backport-PR-URL: #46839Fixes: #46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: #46737 Backport-PR-URL: #46839Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

PR-URL: #46604 Backport-PR-URL: #46839Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

The test runner is bootstrapped synchronously, with the exception of importing custom reporters. To better handle asynchronously throw errors, this commit introduces an internal error type that can be checked for from the test runner's uncaughtException handler. After #46707 and this change land, the other throw statement in the uncaughtException handler can be removed. This will allow the test runner to handle errors thrown from outside of tests (which currently prevents the test runner from reporting results). PR-URL: #46720 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

fossamagnaand others added 5 commits February 25, 2023 23:21

test_runner: avoid swallowing of asynchronously thrown errors
9ec0a4e
Fixes: nodejs#44612 PR-URL: nodejs#45264 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: add reporters
7201845
PR-URL: nodejs#45712Fixes: nodejs#45648 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: report file in test runner events
b628161
PR-URL: nodejs#46030 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. v18.x Issues that can be reproduced on v18.x or PRs targeting the v18.x-staging branch. labels Feb 25, 2023

MoLow mentioned this pull request Feb 25, 2023
test_runner: add initial code coverage support #46017
Merged

MoLow changed the title ~~[v18.x backport] test_runner reporters~~[v18.x backport] test_runner featuresFeb 25, 2023

MoLow force-pushed the backport-v18-test-runner-reporters branch from 985157f to dd5e659Compare February 25, 2023 21:47

cjihrigand others added 14 commits February 25, 2023 23:55

doc,lib,src,test: rename --test-coverage
8ad7668
Add experimental to the name as requested during review. PR-URL: nodejs#46017 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]>

test: fix tap escaping with and without --test
95250e1
PR-URL: nodejs#46311Fixes: nodejs#45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: top-level diagnostics not ommited when running with --test
e1f5a3a
PR-URL: nodejs#46441Fixes: nodejs#45910 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

test_runner: fix missing test diagnostics
f4264f0
PR-URL: nodejs#46450Fixes: nodejs#45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

test: fix tap parser fails if a test logs a number
9c49554
PR-URL: nodejs#46056Fixes: nodejs#46048 Reviewed-By: Moshe Atlow <[email protected]>

test_runner: allow nesting test within describe
1502aca
PR-URL: nodejs#46544Fixes: nodejs#46478 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

test_runner: flatten TAP output when running using --test
f7d8bed
PR-URL: nodejs#46440Fixes: nodejs#45833 Refs: nodejs#45833 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

test_runner: parse non-ascii character correctly
730f11f
PR-URL: nodejs#45736 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

test_runner: display skipped tests in spec reporter output
4a63d24
PR-URL: nodejs#46651 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

test_runner: emit test-only diagnostic warning
b50cfb3
PR-URL: nodejs#46540Fixes: nodejs#46448 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

test_runner: bootstrap reporters before running tests
69100c9
PR-URL: nodejs#46737Fixes: nodejs#46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

test_runner: add describe.only and it.only shorthands
6ef4886
PR-URL: nodejs#46604Fixes: nodejs#46562 Reviewed-By: Moshe Atlow <[email protected]>

MoLow force-pushed the backport-v18-test-runner-reporters branch from dd5e659 to b7aefcbCompare February 25, 2023 21:55

MoLow added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 25, 2023

github-actionsbot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 25, 2023

juanarbol pushed a commit that referenced this pull request Mar 3, 2023
test_runner: parse non-ascii character correctly
8bd3c0f
PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 3, 2023
test_runner: display skipped tests in spec reporter output
638bbe9
PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 3, 2023
test_runner: bootstrap reporters before running tests
faedcde
PR-URL: #46737 Backport-PR-URL: #46839Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 3, 2023
test_runner: add describe.only and it.only shorthands
bbe1f7d
PR-URL: #46604 Backport-PR-URL: #46839Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

juanarbol closed this Mar 3, 2023

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test_runner: avoid swallowing of asynchronously thrown errors
5fdf374
Fixes: #44612 PR-URL: #45264 Backport-PR-URL: #46839 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test: fix tap escaping with and without --test
32b020c
PR-URL: #46311 Backport-PR-URL: #46839Fixes: #45836 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test_runner: fix missing test diagnostics
ba784e8
PR-URL: #46450 Backport-PR-URL: #46839Fixes: #45911 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test: fix tap parser fails if a test logs a number
4f5aff2
PR-URL: #46056 Backport-PR-URL: #46839Fixes: #46048 Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test_runner: parse non-ascii character correctly
894d711
PR-URL: #45736 Backport-PR-URL: #46839 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test_runner: display skipped tests in spec reporter output
0898145
PR-URL: #46651 Backport-PR-URL: #46839 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Moshe Atlow <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test_runner: bootstrap reporters before running tests
28a1317
PR-URL: #46737 Backport-PR-URL: #46839Fixes: #46747 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

juanarbol pushed a commit that referenced this pull request Mar 5, 2023
test_runner: add describe.only and it.only shorthands
0a690ef
PR-URL: #46604 Backport-PR-URL: #46839Fixes: #46562 Reviewed-By: Moshe Atlow <[email protected]>

MoLow deleted the backport-v18-test-runner-reporters branch April 2, 2023 21:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[v18.x backport] test_runner features#46839

[v18.x backport] test_runner features #46839

Uh oh!

MoLow commented Feb 25, 2023•
edited
Loading

Uh oh!

nodejs-github-bot commented Feb 25, 2023

Uh oh!

juanarbol commented Mar 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Uh oh!

[v18.x backport] test_runner features#46839

[v18.x backport] test_runner features #46839

Uh oh!

Conversation

MoLow commented Feb 25, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented Feb 25, 2023

Uh oh!

juanarbol commented Mar 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

MoLow commented Feb 25, 2023•
edited
Loading